Posted by: Radek Wojcik 6 years, 4 months ago

(Comments)

Update your local or S3 stored PyPi mirror or subset index
 

Here is a script I wrote to update your PyPi index stored either locally or on S3:

"""
Job can be run on mirror instance(s) to update local PyPi index
To use with S3, create ~/.boto or set BOTO_CONFIG when running:
[sudo] BOTO_CONFIG=/etc/boto_pypi.cfg python update_pypi.py flask -b your-pypi-s3-bucket
"""
import json
import logging
import os
import uuid
import datetime
from argparse import ArgumentParser

import requests
import BeautifulSoup
import boto
from boto.s3.key import Key

# Local pypi index path, if not using S3
PYPI_PATH = '/centos/pypi/web'

# Send files to S3 rather than downloading them locally, use tmp file
USE_S3 = False
TMP_FILE_FOR_S3 = os.path.join(PYPI_PATH, 'tos3.file.')

# File that tracks when index was last modified
LAST_MODIFIED_FILE = os.path.join(PYPI_PATH, 'last-modified')
S3_LAST_MODIFIED_FILE = 'last-modified'

# e.g. Full path /centos/pypi/web/packages/py2.py3/D/Django
PACKAGE_PATH = os.path.join(PYPI_PATH, 'packages')  # Under that there is a version, and letter i.e d or D

FULL_PACKAGE_PATH = PACKAGE_PATH + '/{python_version}/{first_letter}/{package_name}'
S3_FULL_PACKAGE_PATH = 'packages/{python_version}/{first_letter}/{package_name}'

# Index (simple)
INDEX_PATH = os.path.join(PYPI_PATH, 'simple')

# i.e. /centos/pypi/web/simple/Django/index.html
FULL_INDEX_PATH = INDEX_PATH + '/{package_name}'
S3_FULL_INDEX_PATH = 'simple/{package_name}'

# See the rest via my Gist

See the Gist!

Next I will show you how to set this up via nginx if you're using it locally or via AWS S3 Console for S3 bucket storage.

Currently unrated

Comments