RSSPol

RSS feed generator website with user friendly interface.

Source code based on https://github.com/taroved/pol of an RSS feed generator website with a user friendly interface: enter a page URL, pick the elements that make up a feed, and get a ready-to-use RSS feed. A live deployment is available at feed.dony.me.

Repository layout

pol/                       downloader core (Twisted HTTP server, feed builder, client, db helpers, log, GC monitor)
pol/reader/                feed reader: discovery, parsing, full text, caches, output formats, CLI
frontend/                  Django app (models, views, templates, forms, migrations, fixtures, settings.py.example)
frontend/frontend/assets/  static assets (LESS/SASS, CSS, JS, images)
nginx/                     host nginx template (default.site-example)
tests/                     pytest test suite
.github/workflows/         CI, Docker publish, release, Snyk, Pullfrog
requirements.txt           pinned Python dependencies
Dockerfile                 production image (Python 3.15)
docker-compose.yaml        app + PostgreSQL 19
docker-compose.prod.yml    pull-only production compose (no build)
.env.example               production environment template
.dockerignore
wait-for-it.sh             optional PostgreSQL wait wrapper
downloader.py

Feed reader features

Besides scraping a page into a new feed, pol can read, enrich and re-export existing feeds:

  • Read RSS 0.9x/2.0, Atom 0.3/1.0, RDF (RSS 1.0) and JSON Feed 1.0/1.1.
  • Discover feeds on a page, ported from ashcoft/rss-finder: <link rel="alternate">, feed-looking anchors, well-known paths under the page and then under the site root, and every candidate is verified by parsing it.
  • Fetch the full article text for feed items (readability-style extraction, or a manual XPath).
  • Build feeds from JSON, HTML or XML documents with hand-written rules.
  • Export as RSS 2.0, JSON Feed, CSV, HTML or JSONP.
  • HTTP caching with memory, redis or the built-in disk cache (diskcache backend) and ETag/Last-Modified revalidation.
  • Replace tracking links with direct links, including known redirect wrappers (l.facebook.com, google.com/url, news.google.com, translate.google.*, getpocket.com/redirect, out.reddit.com, ...), feedsportal links and feedburner origLink values.
  • Recover broken encodings (BOM, header, XML declaration, meta charset, legacy Chinese/Japanese/Korean/Cyrillic code pages).
  • Follow HTTP redirects and <meta http-equiv="refresh"> redirects.
  • Same features from a morss-style command line.

The index page accepts ?q=<page url> (for example /?q=https://www.cyclingnews.com) to run discovery immediately. Each result has a Preview action that opens a slide-in pane with the latest feed items, plus Copy and Open actions, mirroring the rss-finder interface.

Reader HTTP API

The downloader server exposes these endpoints (nginx proxies them, see nginx/default.site-example):

EndpointDescription
GET /read?url=<feed>Read a feed, or discover one on an HTML page.
GET /read?url=<feed>&format=json|csv|htmlSame, in another format.
GET /read?url=<feed>&callback=cbJSONP output.
GET /discover?url=<page>{"feeds": [{"title", "url", "type"}]} (rss-finder shape).
GET /api/find-feeds?url=<page>Alias of /discover.
POST /api/find-feedsJSON body {"url": "..."}.
GET /api/feed-preview?url=<feed>Feed metadata and the latest items as JSON.

/read parameters: format, callback, fulltext/fill, proxy, clip, resolve, search, order, firstlink, nolink, noref, xpath, indent, max_items, offline, force, no_meta, plus the custom-feed rules mode, items, item_title, item_link, item_content, item_desc, item_time, item_author, item_image. Full-text fetching is off by default on HTTP and can be enabled with fulltext=1.

The examples use the host nginx URL. In development the downloader alone listens on 1234.

Examples:

curl 'http://localhost/read?url=https://example.com/feed.xml'
curl 'http://localhost/read?url=https://example.com/feed.xml&format=json&fulltext=1'
curl 'http://localhost/discover?url=https://example.com/'

Command line

python -m pol.reader https://example.com/feed.xml --format json --indent
python -m pol.reader https://example.com/ --discover
python -m pol.reader https://example.com/feed.xml --fulltext --resolve --cache \
    --cache-backend diskcache --cache-dir /var/cache/pol
python -m pol.reader https://example.com/api \
    --mode json --items 'data.posts[]' --item-title title \
    --item-link url --item-content body --format rss

Installing the project (pip install .) also provides the morss and pol-feed console scripts.

The morss options are accepted too: --post STRING sends the body to the feed URL, --order first|last|newest|oldest chooses which items are enriched when --max-items is reached (the output order is unchanged), --firstlink takes the item link from the first <a> in the description, --nolink drops anchors from item content while keeping their text, --noref drops item links, and --silent suppresses the output.

Cache backends

Set POL_CACHE_BACKEND (or CACHE_BACKEND in settings.py) to memory (default), diskcache, redis or none. The diskcache backend is a built-in JSON-backed disk cache (no extra package); POL_CACHE_DIR selects its directory. POL_REDIS_URL selects the redis server. The reader uses conditional requests (If-None-Match / If-Modified-Since) and falls back to the memory backend when Redis is unavailable.

Requirements and pinned versions

The production image and CI are built and tested with:

ComponentVersion / requirement
pol1.3.1 (source); production image is latest or POL_VERSION until the next release
Python3.15 (python:3.15-slim)
Django6.1.3
PostgreSQL19 (postgres:19)
nginx1.28.3 (Ubuntu Server 26.04 LTS host package)
Docker Composev2 (required)

The application version is 1.3.1. The 1.3.1 image predates the host-nginx layout, so docker-compose.prod.yml uses ghcr.io/ashcoft/pol:latest until a release containing this change is published. Set POL_VERSION in .env to pin that release tag. docker-compose.yaml also tracks latest for development. Python packages are pinned in requirements.txt. The Dockerfile pins the Python base image tag, and both compose files pin postgres:19. nginx is not in the compose stack: use the Ubuntu Server LTS package on the host (1.28.3 on 26.04) with the included template. Pin the base image digest if you need byte-for-byte reproducible builds. Set POL_VERSION to the new release tag and update this table when a release is published.

System build dependencies for psycopg, lxml and Scrapy, e.g. on Ubuntu/Debian:

sudo apt-get install build-essential pkg-config libpq-dev libxml2-dev libxslt1-dev libffi-dev libssl-dev gettext

Optional, only if you rely on Django Pipeline's LESS/SASS compilation instead of prebuilt assets:

sudo apt-get install nodejs npm
sudo npm install -g less
sudo apt-get install ruby ruby-dev
sudo gem install sass

Development server (Ubuntu)

Set up Python environment and install dependencies(from the repo root):

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
pip install pytest

Create the Django settings module (the repo expects settings.py next to the frontend package; it is git-ignored):

cp frontend/frontend/settings.py.example frontend/frontend/settings.py

Create the database:

createdb pol    # or: psql -U postgres -c 'CREATE DATABASE pol;'

If your local PostgreSQL differs from the settings defaults, export DB_NAME, DB_USER, DB_PASSWORD, DB_HOST and DB_PORT before running the servers. The non-Docker settings defaults are pol / postgres / postgres on 127.0.0.1:5432. .env.example is the Docker template and uses pol / pol / change-me.

Initialise the database:

cd frontend
python manage.py migrate
python manage.py loaddata fields.json
cd ..

(If you have some questions, please contact the author via the GitHub email.)

Run the servers

Run the downloader/RSS server (default port 1234):

python downloader.py

Run the Django frontend:

python frontend/manage.py runserver

For the frontend to reach the downloader (/downloader, /feed, /feed1), install nginx and use the included site template:

sudo apt-get install nginx
sudo cp nginx/default.site-example /etc/nginx/sites-available/pol
sudo ln -s /etc/nginx/sites-available/pol /etc/nginx/sites-enabled/pol
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

The template proxies / to the Django server on port 8000 (Gunicorn in production) and /downloader, /feed, /feed1, /read, /rss, /discover and /api/ to the downloader on port 1234. The Docker stack publishes those same loopback ports for the host nginx.

The template forwards the browser Host header so Django's CSRF checks pass behind the proxy. If you reach the app by anything other than localhost/127.0.0.1, export ALLOWED_HOSTS (comma separated) with that name or IP before starting Django.

Docker

The production image is ghcr.io/ashcoft/pol:latest until the next release. Set POL_VERSION in .env to pin a release tag. docker-compose.yaml also uses latest for development.

latest is rebuilt by the Publish Docker image workflow on every push to main. After the host-nginx change lands, wait for that workflow to finish before pulling latest; otherwise use docker compose up -d --build so the image matches this checkout. The 1.3.1 release predates the host-nginx layout and must not be used with this compose file.

git clone https://github.com/ashcoft/pol.git
cd pol
cp .env.example .env      # optional; edit the values
docker compose up -d

Compose tries to pull ghcr.io/ashcoft/pol:latest first. If the pull succeeds, the image is used and nothing is built. If the pull fails (the package is private, you are not logged in, or the image is not published yet), Compose falls back to building the local Dockerfile because the service also has a build: section. Use docker compose up -d --build to force a local build.

The GHCR package is private by default. Log in once with a token that has read:packages:

echo "$GHCR_TOKEN" | docker login ghcr.io -u <your-github-username> --password-stdin
docker compose pull
docker compose up -d

Set the package visibility to public in the GitHub package settings if you want anonymous pulls.

If you never want the build fallback, use the pull-only production file:

docker compose -f docker-compose.prod.yml up -d

It has no build: section and uses POL_PULL_POLICY (default always), so it pulls the published image and fails explicitly if the registry is unavailable.

If you cannot use GHCR, build the image locally and reuse it with POL_PULL_POLICY=never. Both compose files use the same POL_VERSION, so the local build gets the tag production expects:

POL_VERSION=$(grep -E '^POL_VERSION=' .env 2>/dev/null | cut -d= -f2-)
docker compose up -d --build
POL_PULL_POLICY=never docker compose -f docker-compose.prod.yml up -d

Back up the exact production image so a removed container does not force a GHCR login:

POL_VERSION=$(grep -E '^POL_VERSION=' .env 2>/dev/null | cut -d= -f2-)
docker save "ghcr.io/ashcoft/pol:${POL_VERSION:-latest}" \
  | gzip > /tmp/pol-image.tar.gz

# restore later, then start without pulling
gunzip -c /tmp/pol-image.tar.gz | docker load
POL_PULL_POLICY=never docker compose -f docker-compose.prod.yml up -d

Set POL_PULL_POLICY=never in .env if every production start should use the local image.

The stack starts two services:

  • politepol - ghcr.io/ashcoft/pol:${POL_VERSION:-latest}; Gunicorn and the downloader in one container, running as the unprivileged appuser; publishes 127.0.0.1:8000 and 127.0.0.1:1234 on the host
  • dbpolitepol - postgres:19, with its data in the pol_pg volume mounted at /var/lib/postgresql

The compose stack does not run nginx. Use the nginx from Ubuntu Server LTS (1.28.3 on 26.04) with the included template; it proxies / to 127.0.0.1:8000 and /downloader, /feed, /feed1, /read, /rss, /discover and /api/ to 127.0.0.1:1234.

At start, frontend/start.sh waits for PostgreSQL, runs migrate and loaddata, then starts the downloader and Gunicorn. settings.py is generated at image build and reads the environment at import. collectstatic runs at image build; WhiteNoise serves /static/ through Gunicorn.

Compose fixes DB_HOST=dbpolitepol and DB_PORT=5432; PostgreSQL always listens on 5432 inside the container. The app's settings still read DB_HOST/DB_PORT for non-Docker runs.

PostgreSQL 19 stores its cluster under /var/lib/postgresql/19/docker, so the pol_pg volume is mounted at /var/lib/postgresql, the layout the official image expects. If you have data from an older /var/lib/postgresql/data mount, dump it and restore it into a fresh volume before switching.

Compose reads .env for these values:

VariableDefaultMeaning
DB_NAMEpoldatabase name
DB_USERpoldatabase user
DB_PASSWORDchange-medatabase password
POL_VERSIONlatestapp image tag to pin in production
POL_PULL_POLICYunsetpull policy; production defaults to always, development to missing
TIME_ZONEUTCDjango timezone
DEBUGFalsekeep False in production
SECRET_KEYchange-me-in-productionset a long random value
ALLOWED_HOSTSloopbackcomma-separated public hostnames
CSRF_TRUSTED_ORIGINSemptye.g. https://feeds.example.com
GUNICORN_WORKERS3Gunicorn worker processes
GUNICORN_THREADS4threads per worker
GUNICORN_TIMEOUT120Gunicorn timeout in seconds

Example .env for a public HTTPS site:

DB_PASSWORD=change-me
DEBUG=False
SECRET_KEY=put-a-long-random-value-here
ALLOWED_HOSTS=feeds.example.com
CSRF_TRUSTED_ORIGINS=https://feeds.example.com

Host nginx (Ubuntu Server LTS)

Install nginx from the Ubuntu repository and use the included template:

sudo apt-get install nginx
sudo cp nginx/default.site-example /etc/nginx/sites-available/pol
sudo ln -s /etc/nginx/sites-available/pol /etc/nginx/sites-enabled/pol
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

The template listens on port 80 and proxies to the loopback ports published by Compose. For HTTPS:

sudo apt-get install certbot python3-certbot-nginx
sudo certbot --nginx -d feeds.example.com

Then set ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS to the public HTTPS host.

Operations

# logs
docker compose logs -f politepol

# upgrade
docker compose pull
docker compose up -d

# pull-only production file
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d

# database backup
docker compose exec -T dbpolitepol sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' > pol.sql

# image backup (for hosts that cannot pull from GHCR)
POL_VERSION=$(grep -E '^POL_VERSION=' .env 2>/dev/null | cut -d= -f2-)
docker save "ghcr.io/ashcoft/pol:${POL_VERSION:-latest}" \
  | gzip > /tmp/pol-image.tar.gz
gunzip -c /tmp/pol-image.tar.gz | docker load
POL_PULL_POLICY=never docker compose -f docker-compose.prod.yml up -d

Publishing the image (maintainers)

.github/workflows/docker-publish.yml runs on pushes to main and v* tags:

  • ghcr.io/ashcoft/pol:latest on the default branch
  • ghcr.io/ashcoft/pol:sha-<short> on every push
  • ghcr.io/ashcoft/pol:<version> and <major>.<minor> for v* tags

The package must be linked to the ashcoft/pol repository. If it is not, the workflow fails with denied: permission_denied: read_package. Link it in the package settings (Repository source -> Connect repository), then re-run the workflow. Packages are private by default.

Troubleshooting

SymptomCauseFix
docker compose up starts a build (using docker-compose.yaml)GHCR pull failed (private package, not logged in, or not published)docker login ghcr.io, then docker compose pull; or use --build for a local build
Head ... unauthorized on pullPrivate package and no loginLog in with a read:packages token
permission_denied: read_package in ActionsPackage is not linked to the repositoryLink the package in its settings and re-run the workflow
App waits for the database and exitsPostgreSQL not ready or wrong credentialsdocker compose logs dbpolitepol; check DB_* in .env
dbpolitepol exits with "PostgreSQL data in /var/lib/postgresql/data"Volume created with the old mountDump it first if you need the data, then docker compose down -v and start fresh
Host nginx returns 502App ports not reachabledocker compose ps; the app must publish 127.0.0.1:8000 and 127.0.0.1:1234

Tests

cp frontend/frontend/settings.py.example frontend/frontend/settings.py
pip install pytest
pytest tests/

CI runs pytest -v tests/ on Python 3.15, plus linting, a Docker image build and an import check inside the image. The tests do not need a database; the Django tests load settings only.

End-to-end tests

The Playwright suite runs against a live stack. Start the downloader with the private-host reader flag so the local fixture is reachable:

# terminal 1
POL_READER_ALLOW_PRIVATE=1 python downloader.py 1234

# terminal 2
python -m gunicorn frontend.wsgi:application \
  --pythonpath ./frontend --bind 127.0.0.1:8000

# nginx: copy nginx/default.site-example to sites-available/default and start it

Then:

pip install -r requirements-e2e.txt
python -m playwright install --with-deps chromium
RUN_E2E=1 pytest tests/e2e -v

tests/e2e uses a local HTTP fixture, so it does not call external sites. E2E_BASE_URL selects the URL (default http://127.0.0.1). The E2E workflow starts the full stack and runs the same command; the normal CI run ignores tests/e2e.

License

MIT