[email protected]
[Top] [All Lists]

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for bette

Subject: Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
From: Terry Reedy
Date: Wed, 16 Jun 2010 15:27:09 -0400
On 6/16/2010 8:20 AM, M.-A. Lemburg wrote:
Antoine Pitrou wrote:
Tarek Ziadé<ziade.tarek<at>  gmail.com>  writes:

And we happen to have this network already: lots of people
will host a PyPI mirror as soon as it's easy to set one imho.

You must be careful that the mirrors are properly managed and administered,
though. Having stale/dysfunctioning mirrors is worse than having no mirrors at
It is likely that some people will setup a mirror and then "forget" to take care
about it. Like our buildbots really.

Right, it's that administration overhead I was referring to.

Perhaps we should just let the users decide:

a) they use the default PyPI access (which we then enhance by
    caching the content in the cloud)

b) they setup their easy_install or zc.buildout to pull data
    from a mirror network by enabling a configuration option

Since implementing option b) will require updating existing
package tools on the client side anyway, the extra configuration
shouldn't be a problem.

Option a) requires no changes whatsoever on the client side.

It seems to me that:

If the problem of availability with pypi is anything like the problems with bugs... and extending 'pypi...' with cloud service could be done relatively quickly (within a month), then that seems reasonable.

If 'free to psf' mirrors are feasible and needed, then they will still be useful, especially is high-download regions. Since Amazon's cloud service is metered on a region by region basis, any off-loading of demand to regional mirrors will reduce PSF charges. Based on what I have read in the thread, I would not be surprised if full mirror deployment takes a year. After that, the cloud service could remain to pick up slack in a region should the mirror in a region go down.

Any move to incremental update from time-based replacement will benefit either system.

Terry Jan Reedy

Catalog-SIG mailing list
[email protected]

<Prev in Thread] Current Thread [Next in Thread>