fix logging some more

otherwise it fails to start

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Actual Behavior**
+A clear and concise description of what happens.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Ticketfrei Version**
+See the commit on which Ticketfrei is running at example.org/version.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/something-else.md

@@ -0,0 +1,7 @@
+---
+name: Something else
+about: Other ideas?
+
+---
+
+*If your suggestion is neither a bug report nor a feature request, this is the right place. Just describe what you have in mind.*

LICENSE

@@ -1,6 +1,6 @@
 Copyright (c) 2017 Thomas L <tom@dl6tom.de>
 Copyright (c) 2017 b3yond <b3yond@riseup.net>
-Copyright (c) 2018 sid
+Copyright (c) 2018 sid <sid-sid@riseup.net>
 
 Permission to use, copy, modify, and distribute this software for any
 purpose with or without fee is hereby granted, provided that the above

README.md

@@ -1,10 +1,25 @@
 # Ticketfrei social bot
 
-Version: 2.0beta
-
 Ticketfrei is a mastodon/twitter/mail bot to dodge ticket controllers in public
 transport systems.
 
+## Mission
+
+Public transportation is meant to provide an easy and time-saving way to move
+within a region while being affordable for everybody. Unfortunately, this is
+not yet the case. Ticketfrei's approach is to **enable people to reclaim public
+transportation.**
+
+On short term we want to do this by helping users to avoid controllers and
+fines - on long term by **pressuring public transportation companies to offer
+their services free of charge**, financed by the public.
+
+Because with Ticketfrei you're able to use trains and subways for free anyway.
+Take part and create a new understanding of what public transportation could
+look like!
+
+## How It Works
+
 The functionality is simple: It retweets every tweet where it is mentioned.
 
 This leads to a community which evolves around it. If you see ticket
@@ -13,19 +28,19 @@ your tweet and others can read the info and think twice whether they want to
 buy a ticket or not. If enough people, a critical mass, participate for the bot
 to become reliable, you have positive self-reinforcing dynamics.
 
-Today, you can use a Twitter, a Mastodon, and Mail with the account. They will
-communicate with each other; if someone warns others via Mail, Twitter and
-Mastodon users will also see the message. And vice versa.
+Today, you can use a Twitter, Mastodon, Telegram, and Mail with the account.
+They will communicate with each other; if someone warns others via Mail,
+Telegram, Twitter and Mastodon users will also see the message. And vice versa.
 
-In version 2, this bot has received a frontend website. On this website, people
-can register an own bot for their city - the website manages multiple bots for
-multiple citys. This way, you do not have to host it yourself.
+In version 2, this repository contains a web application. On this website,
+people can register an own bot for their city - the website manages multiple
+bots for multiple citys. This way, you do not have to host it yourself.
 
 In the promotion folder, you'll find some promotion material you can use to
 build up such a community in your city. Unfortunately it is in german - but
 it's editable, feel free to translate it!
 
-Website: https://ticketfrei.links-tech.org
+Website (our flagship instance): https://ticketfrei.links-tech.org
 
 More information: https://wiki.links-tech.org/IT/Ticketfrei
 
@@ -34,9 +49,11 @@ More information: https://wiki.links-tech.org/IT/Ticketfrei
 Just go to https://ticketfrei.links-tech.org or another website where this software is
 running.
 
-* Register a twitter account
-* Register a Mastodon account
 * Register on the ticketfrei site
+* Optionally: register bots:
+  * Register a Twitter account
+  * Register a Mastodon account
+  * Register a Telegram bot
 * Configure account
 * The hard part: do the promotion! You need a community.
 
@@ -50,7 +67,7 @@ to check if something was retweeted in the last hour or something.
 
 To this date, we have never heard of this happening though.
 
-### blockisting
+### Blocklisting
 
 You also need to edit the goodlist and the blocklist. You can do this on the
 website, in the settings of your bot.
@@ -70,9 +87,9 @@ a GitHub issue or write to tech@lists.links-tech.org, we are happy to help and s
 
 We wrote these installation notes, so you can set up the website easily:
 
-### Install
+### Install from the git repository
 
-To Do:
+This guide assumes you are on a Debian 9 Server:
 
 ```shell
 sudo apt install python3 virtualenv uwsgi uwsgi-plugin-python3 nginx git exim4
@@ -91,7 +108,7 @@ virtualenv -p python3 .
 Install the dependencies:
 
 ```shell
-pip install tweepy pytoml Mastodon.py bottle pyjwt pylibscrypt Markdown twx
+pip install tweepy pytoml Mastodon.py bottle pyjwt pylibscrypt Markdown twx gitpython
 ```
 
 Configure the bot:
@@ -101,10 +118,11 @@ cp config.toml.example config.toml
 vim config.toml
 ```
 
-This configuration is only for the admin. Users can log into
+This configuration is only for the admin. Moderators can log into
 twitter/mastodon/mail and configure their personal bot on the settings page.
 
 Set up LetsEncrypt:
+
 ```shell
 sudo apt-get install python-certbot-nginx -t stretch-backports
 sudo certbot --authenticator webroot --installer nginx --agree-tos --redirect --hsts 
@@ -134,10 +152,19 @@ echo "Enter your domain name into the following prompt:" && read DOMAIN
 # configure nginx
 sudo sed -r "s/example.org/$DOMAIN/g" deployment/example.org.conf > /etc/nginx/sites-enabled/$DOMAIN.conf
 
-# create folder for socket & database
+# create folder for database
 sudo mkdir /var/ticketfrei
 sudo chown www-data:www-data -R /var/ticketfrei
 
+# create folder for socket
+sudo mkdir /var/run/ticketfrei
+sudo chown -R www-data:www-data /var/run/ticketfrei
+sudo -s
+echo "mkdir /var/run/ticketfrei" >> /etc/rc.local
+echo "chown -R www-data:www-data /var/run/ticketfrei" >> /etc/rc.local
+echo "service ticketfrei-web restart" >> /etc/rc.local
+exit
+
 # change /etc/aliases permissions to be able to receive reports per mail
 sudo chown root:www-data /etc/aliases
 sudo chmod 664 /etc/aliases
@@ -160,6 +187,17 @@ sudo systemctl daemon-reload
 sudo systemctl start ticketfrei-backend.service
 ```
 
+### Backup
+
+For automated backups, you need to backup these files:
+
+* `/var/ticketfrei/db.sqlite`
+* `/srv/ticketfrei/config.toml`
+* `/etc/aliases`
+
+You can find an example how to do this with borgbackup in the deployment
+folder. Adjust it to your needs.
+
 ### Logs
 
 There are several logfiles which you can look at:
@@ -176,11 +214,15 @@ less /var/log/syslog
 
 # for the nginx web server:
 less /var/log/nginx/example.org_error.log
+
+# for the mail server
+less /var/log/exim4/mainlog
 ```
 
 ### Development Install
 
-If you want to install it locally to develop on it:
+If you want to install it locally to develop on it, note that twitter and mail
+will probably not work. You should test them on a server instead.
 
 ```shell
 sudo apt install python3 virtualenv uwsgi uwsgi-plugin-python3 nginx git
@@ -225,3 +267,113 @@ sudo chown $USER:$USER -R /var/log/ticketfrei
 ./frontend.py & ./backend.py &
 ```
 
+# Project History
+
+## Version 1
+
+- more of less hacked together during a mate-fueled weekend
+- backend-only, twitter & mastodon
+- just a script, which crawled & retweeted tweets, if they match a whitelist & blocklist
+- whitelist & blocklist were just 2 files
+
+## Version 2
+
+Reasons for the rewrite:
+- user management: Users should be able to run a Ticketfrei bot in their city
+  - without needing a server, without needing command line skills
+- more networks; not only Twitter & Mastodon, also Email & Telegram
+
+2 processes: backend & frontend.  
+The two Processes talk via a database.
+The two Processes have separate log files.
+Both processes take some config values from config.toml.
+
+### Backend
+
+The Backend takes care of crawling & spreading the reports.
+
+backend.py:
+- main loop which does the crawling & posting. 
+- loops through all cities in the database
+  - per city it tries all of the networks/bots:
+    - per network/bot it runs the crawl()-function to ask the social network for new reports
+    - then it checks whether the report is appropriate
+    - if yes, it posts the report via all networks/bots, which belong to the city.
+
+config.py: imports config values
+- Imports values from config.toml
+- If there is no config file it tries to use environment variables,
+- Apart from that it uses the default values.
+
+bot.py: bot parent Class
+- just the absolute minimum what a bot needs to be able to do: crawl + post
+- is never instantiated, only inherited from
+
+report.py: report Class
+- defines how reports are supposed to look like
+
+active_bots/mailbot.py as an example for how a network/bot works
+- crawl():
+  - mails arrive at an mbox file through exim4
+  - the bot checks whether they are new
+  - the bot generates a report object from the mail and returns it to the backend.py-loop
+- post():
+  - asks the database for the list of mails which want to receive reports for this city
+  - sends the report.text to those mail addresses
+
+
+### Frontend
+
+the architecture of the frontend is loosely oriented off [Model View
+Controller](https://blog.codinghorror.com/understanding-model-view-controller/).
+
+user.py (Model)
+- high-level interface to talk to the database
+- database calls; almost all values in the database are specific to a city/user
+- user.py is also a Class for frontend web authentication
+- user.py keeps the user-id, through which the frontend tracks authentication
+
+db.py (Model)
+- DB-Layout; creates the database if it doesn't exist yet.
+- holds some database calls which are not city-specific.
+
+frontend.py (Controller): bottle web application
+- handles POST/GET requests
+- talks to the database through user.py
+- everyone can look at the pages, and register
+- but only authenticated users can login and change settings
+
+session.py: User Authentication
+- takes care of session cookies and "403 unauthenticated" error messages
+
+sendmail.py: helper script to send mails
+- sends all mails the frontend, backend, and bots need to send
+
+static/
+- css, images, javascript for the login form etc.
+
+template/ (view)
+- base for the HTML generation, uses the bottle-template-framework
+- wrapper.tpl is the base template for every other template
+
+
+### active_bots: how to implement a new network
+
+If you want to write a new bot, e.g. a Wire-Bot, you have to take these steps:
+
+- look for a python-library which can talk to Wire
+- the city/users have to provide authentication details; this needs a form in
+  the settings
+  - depending on the network either a password, a token, or an implementation
+    of the OAuth login flow
+- the backend needs to crawl messages from the network, & post reports to the
+  network
+
+Files you need to change:
+
+1. active_bots/wire.py - crawl & post functions
+2. settings.tpl - form to authenticate to the network & possible network specific settings.
+3. frontend.py - routes for the forms you added to settings.tpl
+4. db.py - database layout, to store the account credentials/tokens, and to save which message you have last seen
+5. user.py - database calls to get or set values
+

active_bots/mailbot.py

@@ -10,7 +10,7 @@ from bot import Bot
 from config import config
 from db import db
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 class Mailbot(Bot):
@@ -19,7 +19,11 @@ class Mailbot(Bot):
     def crawl(self, user):
         reports = []
         # todo: adjust to actual mailbox
-        mails = mailbox.mbox("/var/mail/" + config['mail']['mbox_user'])
+        try:
+            mails = mailbox.mbox("/var/mail/" + config['mail']['mbox_user'])
+        except FileNotFoundError:
+            logger.error("No mbox file found.")
+            return reports
         for msg in mails:
             if get_date_from_header(msg['Date']) > user.get_seen_mail():
                 if user.get_city().lower() in msg['To'].lower():
@@ -34,7 +38,7 @@ class Mailbot(Bot):
             unsubscribe_text = "\n_______\nYou don't want to receive those messages? Unsubscribe with this link: "
             body = report.text + unsubscribe_text + config['web']['host'] + "/city/mail/unsubscribe/" \
                    + db.mail_subscription_token(rec, user.get_city())
-            if report.author != rec:
+            if rec not in report.author:
                 try:
                     city = user.get_city()
                     sendmail(rec, "Ticketfrei " + city + " Report",
@@ -54,27 +58,16 @@ def make_report(msg, user):
     date = get_date_from_header(msg['Date'])
 
     author = msg['From']  # get mail author from email header
-    # :todo take only the part in between the < >
-
-    if msg.is_multipart():
-        text = []
-        for part in msg.get_payload():
-            if part.get_content_type() == "text":
-                text.append(part.get_payload())
-            elif part.get_content_type() == "multipart/mixed":
-                for p in part:
-                    if p.get_content_type() == "text":
-                        text.append(part.get_payload())
-                    else:
-                        logger.error("unknown MIMEtype: " +
-                                     p.get_content_type())
-            else:
-                logger.error("unknown MIMEtype: " +
-                             part.get_content_type())
-        text = '\n'.join(text)
-    else:
-        text = msg.get_payload()
-    post = report.Report(author, "mail", text, None, date)
+    for part in msg.walk():
+        if part.get_content_type() == 'text/plain':
+            text = part.get_payload()
+        elif part.get_content_type() == 'text/html':
+            text = re.sub(r'<[^>]*>', '', msg.get_payload())
+    try:
+        post = report.Report(author, "mail", text, None, date)
+    except UnboundLocalError:
+        logger.error('No suitable message body')
+        return
     user.save_seen_mail(date)
     return post
 

active_bots/mastodonbot.py

@@ -2,12 +2,12 @@
 
 from bot import Bot
 import logging
-from mastodon import Mastodon
+import mastodon
 import re
 from report import Report
 
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 class MastodonBot(Bot):
@@ -19,42 +19,93 @@ class MastodonBot(Bot):
         """
         mentions = []
         try:
-            m = Mastodon(*user.get_masto_credentials())
+            m = mastodon.Mastodon(*user.get_masto_credentials())
         except TypeError:
-            # logger.error("No Mastodon Credentials in database.", exc_info=True)
+            # No Mastodon Credentials in database.
             return mentions
         try:
             notifications = m.notifications()
-        except Exception:
-            logger.error("Unknown Mastodon API Error.", exc_info=True)
+        except mastodon.MastodonNetworkError:
+            logger.error("Mastodon Network Error.")
+            return mentions
+        except mastodon.MastodonAPIError:
+            try:
+                logger.error("Mastodon API Error: " + m.instance()['urls']['streaming_api'] + ", city: " + str(user.uid))
+            except mastodon.MastodonServerError:
+                logger.error("Mastodon Server Error 500, can't get instance.")
+            except mastodon.MastodonVersionError:
+                logger.error("Mastodon Server Error 500, server version too low.")
+            return mentions
+        except mastodon.MastodonInternalServerError:
+            try:
+                logger.error("Mastodon Error: 500. Server: " + m.instance()['urls']['streaming_api'])
+            except mastodon.MastodonServerError:
+                logger.error("Mastodon Server Error 500, can't get instance.")
+            except mastodon.MastodonVersionError:
+                logger.error("Mastodon Server Error 500, server version too low.")
+            return mentions
+        except mastodon.MastodonBadGatewayError:
+            try:
+                logger.error("Mastodon Error: 502. Server: " + m.instance()['urls']['streaming_api'])
+            except mastodon.MastodonServerError:
+                logger.error("Mastodon Server Error 502, can't get instance.")
+            except mastodon.MastodonVersionError:
+                logger.error("Mastodon Server Error 502, server version too low.")
+            return mentions
+        except mastodon.MastodonServiceUnavailableError:
+            try:
+                logger.error("Mastodon Error: 503. Server: " + m.instance()['urls']['streaming_api'])
+            except mastodon.MastodonServerError:
+                logger.error("Mastodon Server Error 503, can't get instance.")
+            except mastodon.MastodonVersionError:
+                logger.error("Mastodon Server Error 503, server version too low.")
+            return mentions
+        except mastodon.MastodonGatewayTimeoutError:
+            try:
+                logger.error("Mastodon Error: 504. Server: " + m.instance()['urls']['streaming_api'])
+            except mastodon.MastodonServerError:
+                logger.error("Mastodon Server Error 504, can't get instance.")
+            except mastodon.MastodonVersionError:
+                logger.error("Mastodon Server Error 504, server version too low.")
+            return mentions
+        except mastodon.MastodonServerError:
+            try:
+                logger.error("Unknown Mastodon Server Error. Server: " + m.instance()['urls']['streaming_api'], exc_info=True)
+            except mastodon.MastodonServerError:
+                logger.error("Unknown Mastodon Server Error.", exc_info=True)
+            except mastodon.MastodonVersionError:
+                logger.error("Unknown Mastodon Server Error.", exc_info=True)
             return mentions
         for status in notifications:
-            if (status['type'] == 'mention' and
+            try:
+                if (status['type'] == 'mention' and
                     not user.toot_is_seen(status['status']['uri'])):
-                # save state
-                user.toot_witness(status['status']['uri'])
-                # add mention to mentions
-                text = re.sub(r'<[^>]*>', '', status['status']['content'])
-                text = re.sub(
-                    "(?<=^|(?<=[^a-zA-Z0-9-_.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
-                    "", text)
-                if status['status']['visibility'] == 'public':
-                    mentions.append(Report(status['account']['acct'],
-                                           self,
-                                           text,
-                                           status['status']['id'],
-                                           status['status']['created_at']))
-                else:
-                    mentions.append(Report(status['account']['acct'],
-                                           'mastodonPrivate',
-                                           text,
-                                           status['status']['id'],
-                                           status['status']['created_at']))
+                    # save state
+                    user.toot_witness(status['status']['uri'])
+                    # add mention to mentions
+                    text = re.sub(r'<[^>]*>', '', status['status']['content'])
+                    text = re.sub(
+                        "(?<=^|(?<=[^a-zA-Z0-9-_.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
+                        "", text)
+                    if status['status']['visibility'] == 'public':
+                        mentions.append(Report(status['account']['acct'],
+                                               self,
+                                               text,
+                                               status['status']['id'],
+                                               status['status']['created_at']))
+                    else:
+                        mentions.append(Report(status['account']['acct'],
+                                               'mastodonPrivate',
+                                               text,
+                                               status['status']['id'],
+                                               status['status']['created_at']))
+            except TypeError:
+                pass
         return mentions
 
     def post(self, user, report):
         try:
-            m = Mastodon(*user.get_masto_credentials())
+            m = mastodon.Mastodon(*user.get_masto_credentials())
         except TypeError:
             return  # no mastodon account for this user.
         if report.source == self:

active_bots/telegrambot.py

@@ -3,8 +3,7 @@ import logging
 from report import Report
 from twx.botapi import TelegramBot as Telegram
 
-
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 class TelegramBot(Bot):
@@ -13,20 +12,45 @@ class TelegramBot(Bot):
         seen_tg = user.get_seen_tg()
         try:
             updates = tb.get_updates(offset=seen_tg + 1,
-                                     allowed_updates="message").wait()
+                                     allowed_updates="message",
+                                     timeout=5).wait()
         except TypeError:
-            updates = tb.get_updates().wait()
+            updates = tb.get_updates(timeout=5).wait()
         reports = []
         if updates == None:
             return reports
         for update in updates:
             # return when telegram returns an error code
-            if update in [303, 404, 420, 500]:
+            if update in [303, 404, 420, 500, 502]:
                 return reports
-            elif isinstance(update, int):
-                logger.error("Unknown Telegram error code: " + str(update))
+            # log unusual telegram error messages
+            if isinstance(update, int):
+                try:
+                    logger.error("City " + str(user.uid) +
+                                 ": Unknown Telegram error code: " +
+                                 str(update) + " - " + str(updates[1]))
+                except TypeError:
+                    logger.error("Unknown Telegram error code: " + str(update))
                 return reports
+            # save the last message, so it doesn't get crawled again
             user.save_seen_tg(update.update_id)
+            # skip if message is None
+            if update.message is None:
+                continue
+            # complain if message is a photo
+            if update.message.photo is not None:
+                tb.send_message(
+                    update.message.sender.id,
+                    "Sending Photos is not supported for privacy reasons. Can "
+                    "you describe it as text instead?")
+                continue
+            # complain if message is a media file
+            if update.message.text is None:
+                tb.send_message(
+                    update.message.sender.id,
+                    "We only support text reporting for privacy reasons. Can "
+                    "you describe it as text instead?")
+                continue
             if update.message.text.lower() == "/start":
                 user.add_telegram_subscribers(update.message.sender.id)
                 tb.send_message(
@@ -42,19 +66,24 @@ class TelegramBot(Bot):
             elif update.message.text.lower() == "/help":
                 tb.send_message(
                     update.message.sender.id,
-                    "Send reports here to share them with other users. Use /start and /stop to get reports or not.")
+                    "Send reports here to share them with other users. "
+                    "Use /start and /stop to get reports or not.")
                 # TODO: /help message should be set in frontend
             else:
-                reports.append(Report(update.message.sender.username, self,
-                                      update.message.text, None,
-                                      update.message.date))
+                # set report.author to "" to avoid mailbot crash
+                sender_name = update.message.sender.username
+                if sender_name is None:
+                    sender_name = ""
+
+                reports.append(Report(sender_name, self, update.message.text,
+                                      None, update.message.date))
         return reports
 
     def post(self, user, report):
         tb = Telegram(user.get_telegram_credentials())
         text = report.text
         if len(text) > 4096:
-            text = text[:4096 - 4] + u' ...'
+            text = text[:4096 - 2] + " \N{Horizontal ellipsis}"
         try:
             for subscriber_id in user.get_telegram_subscribers():
                 tb.send_message(subscriber_id, text).wait()

active_bots/twitterDMs.py

@@ -1,67 +0,0 @@
-#!/usr/bin/env python3
-
-import logging
-import tweepy
-import re
-import requests
-import report
-import tfglobals
-from time import time
-from bot import Bot
-
-
-logger = logging.getLogger(__name__)
-
-
-class TwitterBot(Bot):
-    def get_api(self, user):
-        keys = user.get_api_keys()
-        auth = tweepy.OAuthHandler(consumer_key=keys[0],
-                                   consumer_secret=keys[1])
-        auth.set_access_token(keys[2],  # access_token_key
-                              keys[3])  # access_token_secret
-        return tweepy.API(auth, wait_on_rate_limit=True)
-
-    def crawl(self, user):
-        """
-        crawls all Tweets which mention the bot from the twitter rest API.
-
-        :return: reports: (list of report.Report objects)
-        """
-        reports = []
-        if tfglobals.last_twitter_request + 60 > time():
-            return reports
-        try:
-            api = self.get_api(user)
-        except IndexError:
-            return reports  # no twitter account for this user.
-        last_dm = user.get_seen_dm()
-        try:
-            if last_dm is None:
-                mentions = api.direct_messages()
-            else:
-                mentions = api.mentions_timeline(since_id=last_dm[0])
-            tfglobals.last_twitter_request = time()
-            for status in mentions:
-                text = re.sub(
-                       "(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
-                       "", status.text)
-                reports.append(report.Report(status.author.screen_name,
-                                             "twitterDM",
-                                             text,
-                                             status.id,
-                                             status.created_at))
-            user.save_seen_dm(last_dm)
-            return reports
-        except tweepy.RateLimitError:
-            logger.error("Twitter API Error: Rate Limit Exceeded",
-                         exc_info=True)
-            # :todo implement rate limiting
-        except requests.exceptions.ConnectionError:
-            logger.error("Twitter API Error: Bad Connection", exc_info=True)
-        except tweepy.TweepError:
-            logger.error("Twitter API Error: General Error", exc_info=True)
-        return []
-
-    def post(self, user, report):
-        pass

active_bots/twitterbot.py

@@ -7,10 +7,9 @@ import requests
 from time import time
 import report
 from bot import Bot
-import tfglobals
 
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 class TwitterBot(Bot):
@@ -30,9 +29,11 @@ class TwitterBot(Bot):
         :return: reports: (list of report.Report objects)
         """
         reports = []
-        #global last_twitter_request
-        if tfglobals.last_twitter_request + 60 > time():
-            return reports
+        try:
+            if user.get_last_twitter_request() + 60 > time():
+                return reports
+        except TypeError:
+            user.set_last_twitter_request(time())
         try:
             api = self.get_api(user)
         except TypeError:
@@ -46,13 +47,12 @@ class TwitterBot(Bot):
                 mentions = api.mentions_timeline()
             else:
                 mentions = api.mentions_timeline(since_id=last_mention)
-            tfglobals.last_twitter_request = time()
+            user.set_last_twitter_request(time())
             for status in mentions:
-                text = re.sub(
-                    "(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
-                    "", status.text)
-                username = "@" + api.me().screen_name
-                if username in status.text:
+                if status._json['in_reply_to_status_id'] == None:
+                    text = re.sub(
+                        "(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
+                        "", status.text)
                     reports.append(report.Report(status.author.screen_name,
                                                  self,
                                                  text,
@@ -67,13 +67,13 @@ class TwitterBot(Bot):
         except requests.exceptions.ConnectionError:
             logger.error("Twitter API Error: Bad Connection", exc_info=True)
         except tweepy.TweepError:
-            logger.error("Twitter API Error: General Error", exc_info=True)
+            logger.error("Twitter API Error: General Error. User: " + str(user.uid), exc_info=True)
         return []
 
     def post(self, user, report):
         try:
             api = self.get_api(user)
-        except IndexError:
+        except TypeError:
             return  # no twitter account for this user.
         try:
             if report.source == self:

backend.py

@@ -5,8 +5,7 @@ from config import config
 from db import db
 import logging
 from sendmail import sendmail
-from time import time
-
+from time import sleep
 
 def shutdown():
     try:
@@ -16,14 +15,15 @@ def shutdown():
     exit(1)
 
 
-last_twitter_request = time()
-
 if __name__ == '__main__':
-    logger = logging.getLogger()
+    logger = logging.getLogger("main")
+    logger.setLevel(logging.DEBUG)
     fh = logging.FileHandler('/var/log/ticketfrei/backend.log')
-    fh.setLevel(logging.DEBUG)
+    formatter = logging.Formatter('%(asctime)s %(levelname)8s: %(message)s')
+    fh.setFormatter(formatter)
     logger.addHandler(fh)
 
+    logger.info("Backend Daemon was started...")
 
     bots = []
     for ActiveBot in active_bots.__dict__.values():
@@ -34,12 +34,14 @@ if __name__ == '__main__':
         while True:
             for user in db.active_users:
                 for bot in bots:
+                    sleep(1)
                     reports = bot.crawl(user)
                     for status in reports:
                         if not user.is_appropriate(status):
                             logger.info("Inaproppriate message: %d %s %s" % (user.uid, status.author, status.text))
                             continue
                         for bot2 in bots:
+                            sleep(1)
                             bot2.post(user, status)
                             logger.info("Resent: %d %s %s" % (user.uid, status.author, status.text))
     except Exception:

config.py

@@ -1,5 +1,70 @@
 import pytoml as toml
+import os
+
+
+def load_env():
+    """
+    load environment variables from the environment. If empty, use default
+    values from config.toml.example.
+
+    :return: config dictionary of dictionaries.
+    """
+    with open('config.toml.example') as defaultconf:
+        configdict = toml.load(defaultconf)
+
+    try:
+        if os.environ['CONSUMER_KEY'] != "":
+            configdict['twitter']['consumer_key'] = os.environ['CONSUMER_KEY']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['CONSUMER_SECRET'] != "":
+            configdict['twitter']['consumer_secret'] = os.environ['CONSUMER_SECRET']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['HOST'] != "":
+            configdict['web']['host'] = os.environ['HOST']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['PORT'] != "":
+            configdict['web']['port'] = os.environ['PORT']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['CONTACT'] != "":
+            configdict['web']['contact'] = os.environ['CONTACT']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['MBOX_USER'] != "":
+            configdict['mail']['mbox_user'] = os.environ['MBOX_USER']
+    except KeyError:
+        pass
+
+    try:
+        if os.environ['DB_PATH'] != "":
+            configdict['database']['db_path'] = os.environ['DB_PATH']
+    except KeyError:
+        pass
+
+    return configdict
+
 
 # read config in TOML format (https://github.com/toml-lang/toml#toml)
-with open('config.toml') as configfile:
-    config = toml.load(configfile)
+try:
+    with open('config.toml') as configfile:
+        config = toml.load(configfile)
+except FileNotFoundError:
+    config = load_env()
+
+if __name__ == "__main__":
+    for category in config:
+        for key in config[category]:
+            print(key + "=" + str(config[category][key]))

config.toml.example

@@ -10,10 +10,7 @@ port = 80
 contact = "b3yond@riseup.net"
 
 [mail]
-mailserver = "smtp.riseup.net"
-user = "user"
-passphrase = "sup3rs3cur3"
-mbox = "root"
+mbox_user = "root"
 
 [database]
 db_path = "/var/ticketfrei/db.sqlite"

db.py

@@ -1,12 +1,13 @@
 from config import config
 import jwt
 import logging
-from os import urandom
+from os import urandom, system
 from pylibscrypt import scrypt_mcf
 import sqlite3
+from time import sleep, time
 
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 class DB(object):
@@ -14,13 +15,25 @@ class DB(object):
         self.conn = sqlite3.connect(dbfile)
         self.cur = self.conn.cursor()
         self.create()
-        self.secret = self.get_secret()
 
     def execute(self, *args, **kwargs):
         return self.cur.execute(*args, **kwargs)
 
     def commit(self):
-        self.conn.commit()
+        start_time = time()
+        while 1:
+            try:
+                self.conn.commit()
+                break
+            except sqlite3.OperationalError as error:
+                # another thread may be writing, give it a chance to finish
+                sleep(0.5)
+                logger.exception()
+                if time() - start_time > 5:
+                    # if it takes this long, something is wrong
+                    system("rcctl restart frontend_daemon")
+                    logger.warning("frontend_daemon is getting restarted")
+                    self.conn.commit()
 
     def close(self):
         self.conn.close()
@@ -115,13 +128,6 @@ class DB(object):
                 FOREIGN KEY(twitter_accounts_id)
                     REFERENCES twitter_accounts(id)
             );
-            CREATE TABLE IF NOT EXISTS telegram_accounts (
-                id          INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
-                user_id     INTEGER,
-                api_token   TEXT,
-                active      INTEGER,
-                FOREIGN KEY(user_id) REFERENCES user(id)
-            );
             CREATE TABLE IF NOT EXISTS telegram_subscribers (
                 id          INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
                 user_id     INTEGER,
@@ -141,6 +147,12 @@ class DB(object):
                 mail_date   REAL,
                 FOREIGN KEY(user_id) REFERENCES user(id)
             );
+            CREATE TABLE IF NOT EXISTS twitter_last_request (
+                id          INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
+                user_id     INTEGER,
+                date        INTEGER,
+                FOREIGN KEY(user_id) REFERENCES user(id)
+            );
             CREATE TABLE IF NOT EXISTS cities (
                 id          INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
                 user_id     INTEGER,
@@ -190,7 +202,7 @@ class DB(object):
             'passhash': scrypt_mcf(
                 password.encode('utf-8')
             ).decode('ascii')
-        }, self.secret).decode('ascii')
+        }, self.get_secret()).decode('ascii')
 
     def mail_subscription_token(self, email, city):
         """
@@ -204,17 +216,17 @@ class DB(object):
         token = jwt.encode({
             'email': email,
             'city': city
-        }, self.secret).decode('ascii')
+        }, self.get_secret()).decode('ascii')
         return token
 
     def confirm_subscription(self, token):
-        json = jwt.decode(token, self.secret)
+        json = jwt.decode(token, self.get_secret())
         return json['email'], json['city']
 
     def confirm(self, token, city):
         from user import User
         try:
-            json = jwt.decode(token, self.secret)
+            json = jwt.decode(token, self.get_secret())
         except jwt.DecodeError:
             return None  # invalid token
         if 'passhash' in json.keys():
@@ -246,17 +258,19 @@ u\d\d?"""
         else:
             uid = json['uid']
         with open("/etc/aliases", "a+") as f:
-            f.write(city + ": " + config["mail"]["mbox_user"])
+            f.write(city + ": " + config["mail"]["mbox_user"] + "\n")
+            try:
+                os.system("newaliases")
+            except:
+                logger.exception()
         self.execute("INSERT INTO email (user_id, email) VALUES(?, ?);",
                      (uid, json['email']))
         self.execute("""INSERT INTO telegram_accounts (user_id, apikey,
                         active) VALUES(?, ?, ?);""", (uid, "", 1))
-        self.execute(
-            "INSERT INTO seen_telegrams (user_id, tg_id) VALUES (?, ?);", (uid, 0))
-        self.execute(
-            "INSERT INTO seen_mail (user_id, mail_date) VALUES (?, ?);", (uid, 0))
-        self.execute("INSERT INTO seen_tweets (user_id, tweet_id) VALUES (?, ?)",
-                     (uid, 0))
+        self.execute("INSERT INTO seen_telegrams (user_id, tg_id) VALUES (?, ?);", (uid, 0))
+        self.execute("INSERT INTO seen_mail (user_id, mail_date) VALUES (?, ?);", (uid, 0))
+        self.execute("INSERT INTO seen_tweets (user_id, tweet_id) VALUES (?, ?)", (uid, 0))
+        self.execute("INSERT INTO twitter_last_request (user_id, date) VALUES (?, ?)", (uid, 0))
         self.commit()
         user = User(uid)
         user.set_city(city)

deployment/backend_daemon

@@ -0,0 +1,15 @@
+#!/bin/ksh
+
+daemon="/usr/local/bin/python3 /srv/ticketfrei/backend.py"
+daemon_user="root"
+
+. /etc/rc.d/rc.subr
+
+rc_bg=YES
+rc_reload=NO
+
+rc_start() {
+        rc_exec "cd /srv/ticketfrei; /usr/bin/nice -n15 ${daemon} ${daemon_flags}"
+}
+
+rc_cmd $1

deployment/borgbackup.sh

@@ -0,0 +1,15 @@
+#!/bin/ksh
+. /etc/borg-env
+export BORG_REPO=nathan@nephilim:repositories-borg/ticketfrei
+export BORG_RSH="ssh \
+-o TCPKeepAlive=no \
+-o ServerAliveInterval=15 \
+-o ServerAliveCountMax=10 \
+-o Compression=no"
+
+rcctl stop backend_daemon
+rcctl stop frontend_daemon
+/usr/local/bin/borg create --stats ::'backup{now:%Y%m%d-%H%M}' /srv/ticketfrei /var/ticketfrei /etc
+rcctl start backend_daemon
+rcctl start frontend_daemon
+

deployment/example.org.conf

@@ -1,31 +0,0 @@
-server {
-
-    listen 443 ssl;
-    server_name example.org;
-    ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;
-    ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;
-    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
-    ssl_ciphers         HIGH:!aNULL:!MD5;
-
-    access_log /var/log/nginx/example.org_access.log;
-    error_log /var/log/nginx/example.org_error.log;
-
-    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
-
-    location / {
-        include uwsgi_params;
-
-        uwsgi_pass unix:///var/run/ticketfrei/ticketfrei.sock;
-    }
-
-    location /.well-known/acme-challenge {
-        root /var/www/acme;
-    }
-}
-server {
-    listen 80;
-    listen [::]:80;
-    server_name example.org;
-    return 301 https://$server_name$request_uri;
-}
-

deployment/frontend_daemon

@@ -0,0 +1,15 @@
+#!/bin/ksh
+
+daemon="/usr/local/bin/python3 /srv/ticketfrei/frontend.py"
+daemon_user="frontend"
+
+. /etc/rc.d/rc.subr
+
+rc_bg=YES
+rc_reload=NO
+
+rc_start() {
+        rc_exec "env > /tmp/envars; cd /srv/ticketfrei; ${daemon} ${daemon_flags}"
+}
+
+rc_cmd $1

deployment/ticketfrei-backend.service

@@ -1,17 +0,0 @@
-[Unit]
-Description=Ticketfrei Backend
-After=syslog.target network.target
-
-[Service]
-WorkingDirectory=/srv/ticketfrei
-ExecStart=/srv/ticketfrei/bin/python3 backend.py
-# Requires systemd version 211 or newer
-#RuntimeDirectory=uwsgi
-Restart=always
-KillSignal=SIGQUIT
-Type=simple
-StandardError=syslog
-NotifyAccess=all
-
-[Install]
-WantedBy=multi-user.target

deployment/ticketfrei-web.service

@@ -1,17 +0,0 @@
-[Unit]
-Description=Ticketfrei Web Application
-After=syslog.target network.target
-
-[Service]
-WorkingDirectory=/srv/ticketfrei
-ExecStart=/usr/bin/uwsgi --ini /srv/ticketfrei/deployment/uwsgi.ini
-# Requires systemd version 211 or newer
-RuntimeDirectory=uwsgi
-Restart=always
-KillSignal=SIGQUIT
-Type=notify
-StandardError=syslog
-NotifyAccess=all
-
-[Install]
-WantedBy=multi-user.target

frontend.py

@@ -45,7 +45,7 @@ def register_post():
         sendmail(
                 email,
                 "Confirm your account",
-                "Complete your registration here: %s" % (link)
+                body="Complete your registration here: %s" % (link)
             )
         return dict(info='Confirmation mail sent.')
     except Exception:
@@ -56,11 +56,22 @@ def register_post():
 @get('/confirm/<city>/<token>')
 @view('template/propaganda.tpl')
 def confirm(city, token):
+    # check whether city already exists
+    if db.by_city(city):
+        return dict(error='This Account was already confirmed, please try '
+                          'signing in.')
     # create db-entry
     if db.confirm(token, city):
         # :todo show info "Account creation successful."
         redirect('/settings')
-    return dict(error='Email confirmation failed.')
+    return dict(error='Account creation failed. Please try to register again.')
+
+
+@get('/version')
+def version():
+    import git
+    repo = git.Repo(search_parent_directories=True)
+    return repo.head.object.hexsha
 
 
 @post('/login')
@@ -105,7 +116,7 @@ def subscribe_mail(city):
     # send mail with code to email
     sendmail(email, "Subscribe to Ticketfrei " + city + " Mail Notifications",
              body="To subscribe to the mail notifications for Ticketfrei " +
-                  city + ", click on this link: " + confirm_link)
+                  city + ", click on this link: " + confirm_link, city=city)
     return city_page(city, info="Thanks! You will receive a confirmation mail.")
 
 
@@ -167,9 +178,10 @@ def register_telegram(user):
     return city_page(user.get_city(), info="Thanks for registering Telegram!")
 
 
-@get('/api/state')
-def api_enable(user):
-    return user.state()
+# unused afaik
+#@get('/api/state')
+#def api_enable(user):
+#    return user.state()
 
 
 @get('/static/<filename:path>')
@@ -186,6 +198,7 @@ def guides(filename):
 def logout():
     # clear auth cookie
     response.set_cookie('uid', '', expires=0, path="/")
+    response.set_cookie('csrf', '', expires=0, path="/")
     # :todo show info "Logout successful."
     redirect('/')
 
@@ -244,11 +257,6 @@ def login_mastodon(user):
     try:
         access_token = m.log_in(masto_email, masto_pass)
         user.save_masto_token(access_token, instance_url)
-
-        # Trying to set the seen_toot to 0, thereby initializing it.
-        # It should work now, but has default values. Not sure if I need them.
-        user.init_seen_toot(instance_url)
-
         return city_page(user.get_city(), info='Thanks for supporting decentralized social networks!')
     except Exception:
         logger.error('Login to Mastodon failed.', exc_info=True)
@@ -264,7 +272,6 @@ application = bottle.default_app()
 bottle.install(SessionPlugin('/'))
 
 if __name__ == '__main__':
-    # testing only
-    bottle.run(host=config["web"]["host"], port=config["web"]["port"])
+    bottle.run(host="0.0.0.0", port=config["web"]["port"])
 else:
     application.catchall = False

guides/use.md

@@ -1,43 +0,0 @@
-# How to use Ticketfrei
-
-Do you want to help sousveilling ticket controllers?
-
-## Do you want to know if it's safe to ride without a ticket at the moment?
-
-Just look at the profile of the bot: https://chaos.social/@nbg_ticketfrei
-
-Do you see a toot, reporting ticket controllers?
-* If yes, you should probably buy a ticket for now.
-In Nuremberg we made the experience that ticket controllers are usually active for about a week, a few hours every day.
-So if you see that there was a warning in the last days, watch out.
-* If no, you are probably fine! Dare to ride without a ticket.
-
-We can't guarantee that you will be safe though, so still watch out.
-The more people participate, the more you can trust that controllers are reported before you run into them.
-So, if you have bad luck and are the first one to see the controller:
-
-## Do you want to help others, who ride public transport without a ticket?
-
-That's easy. You only need an Mastodon account, for example at
-* https://queer.party/about
-* https://soc.ialis.me/about
-* https://witches.town/about
-* https://kitty.town/about
-* https://social.coop/about
-* https://awoo.space/about
-
-Using a twitter account or sending a mail is also possible!
-
-Just write a toot or a tweet, mentioning the bot, and tell it
-* Where you saw the ticket controllers
-* Which line they are using, into which direction
-
-For example like this:
-
-![Screenshot of tooting](tooting_screenshot.png)
-
-![A toot ready to be boosted](toot_screenshot.png)
-
-The bot will soon share your message, so other people will be able to look at it and be safe.
-
-Thanks for helping to provide public transport for everyone!

logs/.gitignore

@@ -1,4 +0,0 @@
-# Ignore everything in this directory
-*
-# Except this file
-!.gitignore

requirements.txt

@@ -0,0 +1,9 @@
+tweepy
+pytoml
+Mastodon.py
+bottle
+pyjwt
+pylibscrypt
+Markdown
+twx
+gitpython

sendmail.py

@@ -8,7 +8,7 @@ import smtplib
 from socket import getfqdn
 
 
-logger = logging.getLogger(__name__)
+logger = logging.getLogger("main")
 
 
 def sendmail(to, subject, city=None, body=''):
@@ -27,5 +27,5 @@ def sendmail(to, subject, city=None, body=''):
 
 # For testing:
 if __name__ == '__main__':
-    sendmail(config['mail']['contact'], "Test Mail",
+    sendmail(config['web']['contact'], "Test Mail",
              body="This is a test mail.")

session.py

@@ -1,4 +1,4 @@
-from bottle import redirect, request
+from bottle import redirect, request, abort, response
 from db import db
 from functools import wraps
 from inspect import Signature
@@ -17,10 +17,14 @@ class SessionPlugin(object):
         if self.keyword in Signature.from_callable(route.callback).parameters:
             @wraps(callback)
             def wrapper(*args, **kwargs):
-                uid = request.get_cookie('uid', secret=db.secret)
+                uid = request.get_cookie('uid', secret=db.get_secret())
                 if uid is None:
                     return redirect(self.loginpage)
                 kwargs[self.keyword] = User(uid)
+                if request.method == 'POST':
+                    if request.forms['csrf'] != request.get_cookie('csrf',
+                                                        secret=db.get_secret()):
+                        abort(400)
                 return callback(*args, **kwargs)
 
             return wrapper

static/css/style.css

@@ -1,6 +1,6 @@
 body {
   background-image: url(/static/img/ticketfrei-og-image.jpg);
-  background-size: 50%;
+  background-height: 100%;
   font-family: Verdana, Arial, Helvetica, sans-serif;
   font-size: 12pt;
   line-height: 1.5em;

template/propaganda.tpl

@@ -15,13 +15,14 @@
     your city into a paradise for fare dodgers.
 </p>
 <p>
-    Ticketfrei is a Twitter, Mastodon, and E-Mail bot. Users
-    can help each other by tweeting, tooting, or mailing,
-    when and where they spot a ticket controller.
+    Ticketfrei is a Twitter, Mastodon, Telegram, and E-Mail 
+    bot. Users can help each other by tweeting, tooting, 
+    messaging, or mailing, when and where they spot a ticket 
+    controller.
 </p>
 <p>
-    Ticketfrei automatically retweets, boosts, and remails
-    those controller reports, so others can see them. If there
+    Ticketfrei automatically spreads those controller reports 
+    in the other networks, so others can see them. If there
     are ticket controllers around, they can still buy a ticket
     - but if the coast is clear, they can save the money.
 </p>
@@ -31,22 +32,26 @@
     to other citys. There are four basic steps:
 </p>
 <ul>
-    <li>Create a Twitter and/or a Mastodon account.</li>
     <li>Register on this website to create a bot for your city.</li>
+    <li>Create a Twitter, a Telegram, and/or a Mastodon account.</li>
     <li>Log in with the social media accounts you want to
     use for Ticketfrei.</li>
     <li>Promote the service! Ticketfrei only works if there is
     a community for it. Fortunately, we prepared some material
-    you can use:
-    <a href="https://github.com/ticketfrei/promotion" target="_blank">https://github.com/ticketfrei/promotion</a></li>
+    you can edit, remix, use, and republish:
+    <a href="https://github.com/ticketfrei/promotion" target="_blank">https://github.com/ticketfrei/promotion</a>
+    <ul>
+        <li>If you build cool promotion material yourself, please
+        share it with us, so others can use it, too!</li>
+    </ul></li>
 </ul>
 % include('template/register-plain.tpl')
 <h2>Our Mission</h2>
 <p>
     Public transportation is meant to provide an easy and
     time-saving way to move within a region while being
-    affordable for everybody. Unfortunately, this is not the
-    case. Ticketfrei's approach is to enable people to
+    affordable for everybody. Unfortunately, this is not yet
+    the case. Ticketfrei's approach is to enable people to
     reclaim public transportation.
 </p>
 <p>
@@ -58,7 +63,7 @@
 <p>
     Because with Ticketfrei you're able to use trains and
     subways for free anyway. Take part and create a new
-    understanding of what public transportation should look
+    understanding of what public transportation could look
     like!
 </p>
 

template/settings.tpl

@@ -61,6 +61,7 @@
                 <option value='octodon.social'>
                 <option value='soc.ialis.me'>
             </datalist>
+            <input name='csrf' value='{{csrf}}' type='hidden' />
             <input name='confirm' value='Log in' type='submit'/>
         </form>
 </section>
@@ -82,6 +83,7 @@
     </p>
     <form action="/settings/telegram" method="post">
         <input type="text" name="apikey" placeholder="Telegram bot API key" id="apikey">
+        <input name='csrf' value='{{csrf}}' type='hidden' />
         <input name='confirm' value='Login with Telegram' type='submit'/>
     </form>
 </div>
@@ -106,6 +108,7 @@
     </p>
     <form action="/settings/markdown" method="post">
         <textarea id="markdown" rows="20" cols="70" name="markdown" wrap="physical">{{markdown}}</textarea>
+        <input name='csrf' value='{{csrf}}' type='hidden' />
         <input name='confirm' value='Save' type='submit'/>
     </form>
 </div>
@@ -123,6 +126,7 @@
     </p>
     <form action="/settings/mail_md" method="post">
         <textarea id="mail_md" rows="20" cols="70" name="mail_md" wrap="physical">{{mail_md}}</textarea>
+        <input name='csrf' value='{{csrf}}' type='hidden' />
         <input name='confirm' value='Save' type='submit'/>
     </form>
 </div>
@@ -137,6 +141,7 @@
     </p>
     <form action="/settings/goodlist" method="post">
         <textarea id="goodlist" rows="8" cols="70" name="goodlist" wrap="physical">{{triggerwords}}</textarea>
+        <input name='csrf' value='{{csrf}}' type='hidden' />
         <input name='confirm' value='Submit' type='submit'/>
     </form>
 </div>
@@ -151,6 +156,7 @@
     </p>
     <form action="/settings/blocklist" method="post">
         <textarea id="blocklist" rows="8" cols="70" name="blocklist" wrap="physical">{{badwords}}</textarea>
+        <input name='csrf' value='{{csrf}}' type='hidden' />
         <input name='confirm' value='Submit' type='submit'/>
     </form>
 </div>

tfglobals.py

@@ -1,10 +0,0 @@
-from time import time
-
-"""
-This file is for shared global variables. They only stay during runtime.
-
-For reference: 
-https://stackoverflow.com/questions/15959534/visibility-of-global-variables-in-imported-modules
-"""
-
-last_twitter_request = time()

user.py

@@ -1,16 +1,26 @@
 from config import config
-from bottle import response
+from bottle import response, request
 from db import db
 import jwt
 from mastodon import Mastodon
 from pylibscrypt import scrypt_mcf, scrypt_mcf_check
+from os import urandom
+import logging
 
+logger = logging.getLogger("main")
 
 class User(object):
     def __init__(self, uid):
         # set cookie
-        response.set_cookie('uid', uid, secret=db.secret, path='/')
+        response.set_cookie('uid', uid, secret=db.get_secret(), path='/')
         self.uid = uid
+        response.set_cookie('csrf', self.get_csrf(), db.get_secret(), path='/')
+
+    def get_csrf(self):
+        csrf_token = request.get_cookie('csrf', secret=db.get_secret())
+        if not csrf_token:
+            csrf_token = str(urandom(32))
+        return csrf_token
 
     def check_password(self, password):
         db.execute("SELECT passhash FROM user WHERE id=?;", (self.uid,))
@@ -55,7 +65,7 @@ class User(object):
         return jwt.encode({
             'email': email,
             'uid': self.uid
-        }, db.secret).decode('ascii')
+        }, db.get_secret()).decode('ascii')
 
     def is_appropriate(self, report):
         db.execute("SELECT patterns FROM triggerpatterns WHERE user_id=?;",
@@ -66,6 +76,7 @@ class User(object):
                 break
         else:
             # no pattern matched
+            logger.error("Message didn't trigger goodlist: " + report.text)
             return False
         default_badwords = """
 bitch
@@ -85,25 +96,38 @@ schlitz
         db.execute("SELECT words FROM badwords WHERE user_id=?;",
                    (self.uid, ))
         badwords = db.cur.fetchone()
-        for word in report.text.lower().splitlines():
-            if word in badwords:
+        for word in report.text.lower().split():
+            if word in badwords[0].splitlines():
+                logger.error("Word " + word + " triggered the spam filter on message: " + report.text)
                 return False
-        for word in default_badwords.splitlines():
-            if word in badwords:
+        for word in report.text.lower().split():
+            if word in default_badwords.splitlines():
+                logger.error("Word " + word + " triggered the spam filter on message: " + report.text)
                 return False
+        logger.info("Valid report: " + report.text + " | username: " + report.author)
         return True
 
+    def get_last_twitter_request(self):
+        db.execute("SELECT date FROM twitter_last_request WHERE user_id = ?;",
+                   (self.uid,))
+        return db.cur.fetchone()[0]
+
+    def set_last_twitter_request(self, date):
+        db.execute("UPDATE twitter_last_request SET date = ? WHERE user_id = ?;",
+                   (date, self.uid))
+        db.commit()
+
     def get_telegram_credentials(self):
-        db.execute("""SELECT apikey 
-                          FROM telegram_accounts 
+        db.execute("""SELECT apikey
+                          FROM telegram_accounts
                           WHERE user_id = ? AND active = 1;""",
                    (self.uid,))
         row = db.cur.fetchone()
         return row[0]
 
     def get_telegram_subscribers(self):
-        db.execute("""SELECT subscriber_id 
-                          FROM telegram_subscribers 
+        db.execute("""SELECT subscriber_id
+                          FROM telegram_subscribers
                           WHERE user_id = ?;""",
                    (self.uid,))
         rows = db.cur.fetchall()
@@ -116,21 +140,21 @@ schlitz
         db.commit()
 
     def remove_telegram_subscribers(self, subscriber_id):
-        db.execute("""DELETE 
-                          FROM telegram_subscribers 
+        db.execute("""DELETE
+                          FROM telegram_subscribers
                           WHERE user_id = ?
                           AND subscriber_id = ?;""",
                    (self.uid, subscriber_id))
         db.commit()
 
     def get_masto_credentials(self):
-        db.execute("""SELECT access_token, instance_id 
-                          FROM mastodon_accounts 
+        db.execute("""SELECT access_token, instance_id
+                          FROM mastodon_accounts
                           WHERE user_id = ? AND active = 1;""",
                    (self.uid,))
         row = db.cur.fetchone()
-        db.execute("""SELECT instance, client_id, client_secret 
-                          FROM mastodon_instances 
+        db.execute("""SELECT instance, client_id, client_secret
+                          FROM mastodon_instances
                           WHERE id = ?;""",
                    (row[1],))
         instance = db.cur.fetchone()
@@ -225,6 +249,7 @@ schlitz
         # - mail_md
         # - goodlist
         # - blocklist
+        # - csrf
         # - logged in with twitter?
         # - logged in with mastodon?
         # - enabled?
@@ -234,7 +259,8 @@ schlitz
                     mail_md=citydict['mail_md'],
                     triggerwords=self.get_trigger_words(),
                     badwords=self.get_badwords(),
-                    enabled=self.enabled)
+                    enabled=self.enabled,
+                    csrf=self.get_csrf())
 
     def save_request_token(self, token):
         db.execute("""INSERT INTO
@@ -246,11 +272,11 @@ schlitz
         db.commit()
 
     def get_request_token(self):
-        db.execute("""SELECT request_token, request_token_secret 
-                          FROM twitter_request_tokens 
+        db.execute("""SELECT request_token, request_token_secret
+                          FROM twitter_request_tokens
                           WHERE user_id = ?;""", (self.uid,))
         request_token = db.cur.fetchone()
-        db.execute("""DELETE FROM twitter_request_tokens 
+        db.execute("""DELETE FROM twitter_request_tokens
                           WHERE user_id = ?;""", (self.uid,))
         db.commit()
         return {"oauth_token": request_token[0],
@@ -261,6 +287,8 @@ schlitz
                            user_id, client_id, client_secret
                            ) VALUES(?, ?, ?);""",
                    (self.uid, access_token, access_token_secret))
+        db.execute("""INSERT INTO seen_tweets(user_id, tweet_id) VALUES (?, ?);""",
+                   (self.uid, 0))
         db.commit()
 
     def get_twitter_token(self):
@@ -281,8 +309,8 @@ schlitz
         db.commit()
 
     def get_mastodon_app_keys(self, instance):
-        db.execute("""SELECT client_id, client_secret 
-                          FROM mastodon_instances 
+        db.execute("""SELECT client_id, client_secret
+                          FROM mastodon_instances
                           WHERE instance = ?;""", (instance,))
         try:
             row = db.cur.fetchone()
@@ -290,7 +318,7 @@ schlitz
             client_secret = row[1]
             return client_id, client_secret
         except TypeError:
-            app_name = "ticketfrei" + str(db.secret)[0:4]
+            app_name = "ticketfrei" + str(db.get_secret())[0:4]
             client_id, client_secret \
                 = Mastodon.create_app(app_name, api_base_url=instance)
             db.execute("""INSERT INTO mastodon_instances(
@@ -301,8 +329,8 @@ schlitz
             return client_id, client_secret
 
     def save_masto_token(self, access_token, instance):
-        db.execute("""SELECT id 
-                          FROM mastodon_instances 
+        db.execute("""SELECT id
+                          FROM mastodon_instances
                           WHERE instance = ?;""", (instance,))
         instance_id = db.cur.fetchone()[0]
         db.execute("INSERT INTO mastodon_accounts(user_id, access_token, instance_id, active) "
@@ -338,25 +366,28 @@ Schau einfach auf das Profil unseres Bots: """ + twit_link + """
 
 Hat jemand vor kurzem etwas über Kontrolleur\*innen gepostet?
 
-* Wenn ja, dann kauf dir vllt lieber ein Ticket. In Nürnberg 
+* Wenn ja, dann kauf dir vllt lieber ein Ticket. In Nürnberg
   haben wir die Erfahrung gemacht, dass Kontis normalerweile
-  ungefähr ne Woche aktiv sind, ein paar Stunden am Tag. Wenn es 
-  also in den letzten Stunden einen Bericht gab, pass lieber 
+  ungefähr ne Woche aktiv sind, ein paar Stunden am Tag. Wenn es
+  also in den letzten Stunden einen Bericht gab, pass lieber
   auf.
 * Wenn nicht, ist es wahrscheinlich kein Problem :)
 
-Wir können natürlich nicht garantieren, dass es sicher ist, 
+Wir können natürlich nicht garantieren, dass es sicher ist,
 also pass trotzdem auf, wer auf dem Bahnsteig steht.
-Aber je mehr Leute mitmachen, desto eher kannst du dir sicher 
+Aber je mehr Leute mitmachen, desto eher kannst du dir sicher
 sein, dass wir sie finden, bevor sie uns finden.
 
 Wenn du immer direkt gewarnt werden willst, kannst du auch die
-Benachrichtigungen über E-Mail oder Telegram aktivieren. Gib 
-einfach <a href="/city/mail/""" + city + """"/">hier</a> deine 
-E-Mail-Adresse an oder subscribe dem Telegram-Bot [@ticketfrei_""" + city + \
-                   "_bot](https://t.me/ticketfrei_" + city + """_bot) 
+Benachrichtigungen über E-Mail, Telegram, oder den Mastodon RSS
+feed aktivieren. Entweder:
+* Gibt hier [deine E-Mail-Adresse an](/city/mail/""" + city + """)
+* Subscribe dem Telegram-Bot [@ticketfrei_""" + city + \
+"_bot](https://t.me/ticketfrei_" + city + """_bot)
+* oder subscribe dem RSS feed von [""" + city + """](""" + masto_link + \
+""".atom?replies=false&boosts=true)
 
-Also, wenn du weniger Glück hast, und der erste bist, der einen 
+Also, wenn du weniger Glück hast, und der erste bist, der einen
 Kontrolleur sieht:
 
 ## Was mache ich, wenn ich Kontis sehe?
@@ -367,10 +398,10 @@ Ganz einfach, du schreibst es den anderen. Das geht entweder
 * über Twitter: [Link zu unserem Profil](""" + twit_link + """)
 * über Telegram an [@ticketfrei_""" + city + "_bot](https://t.me/ticketfrei_" \
                    + city + """_bot)
-* Oder per Mail an [""" + mailinglist + "](mailto:" + mailinglist + """), wenn 
+* Oder per Mail an [""" + mailinglist + "](mailto:" + mailinglist + """), wenn
   ihr kein Social Media benutzen wollt.
 
-Schreibe einfach einen Toot oder einen Tweet, der den Bot 
+Schreibe einfach einen Toot oder einen Tweet, der den Bot
 mentioned, und gib an
 
 * Wo du die Kontis gesehen hast
@@ -378,15 +409,15 @@ mentioned, und gib an
 
 Zum Beispiel so:
 
-![Screenshot of writing a Toot](https://github.com/b3yond/ticketfrei/raw/master/guides/tooting_screenshot.png)
+![Screenshot of writing a Toot](https://github.com/b3yond/ticketfrei/raw/stable1/guides/tooting_screenshot.png)
 
-![A toot ready to be shared](https://github.com/b3yond/ticketfrei/raw/master/guides/toot_screenshot.png)
+![A toot ready to be shared](https://github.com/b3yond/ticketfrei/raw/stable1/guides/toot_screenshot.png)
 
-Der Bot wird die Nachricht dann weiterverbreiten, auch zu den 
+Der Bot wird die Nachricht dann weiterverbreiten, auch zu den
 anderen Netzwerken.
 Dann können andere Leute das lesen und sicher vor Kontis sein.
 
-Danke, dass du mithilfst, öffentlichen Verkehr für alle 
+Danke, dass du mithilfst, öffentlichen Verkehr für alle
 sicherzustellen!
 
 ## Kann ich darauf vertrauen, was random stranger from the Internet mir da erzählen?
@@ -395,31 +426,31 @@ Aber natürlich! Wir haben Katzenbilder!
 
 ![Katzenbilder!](https://lorempixel.com/550/300/cats)
 
-Glaubt besser nicht, wenn jemand postet, dass die Luft da und 
+Glaubt besser nicht, wenn jemand postet, dass die Luft da und
 da gerade rein ist.
-Das ist vielleicht sogar gut gemeint - aber klar könnte die 
+Das ist vielleicht sogar gut gemeint - aber klar könnte die
 VAG sich hinsetzen und einfach lauter Falschmeldungen posten.
 
-Aber Falschmeldungen darüber, dass gerade Kontis i-wo unterwegs 
+Aber Falschmeldungen darüber, dass gerade Kontis i-wo unterwegs
 sind?
-Das macht keinen Sinn. 
-Im schlimmsten Fall kauft jmd mal eine Fahrkarte mehr - aber 
+Das macht keinen Sinn.
+Im schlimmsten Fall kauft jmd mal eine Fahrkarte mehr - aber
 kann sonst immer schwarz fahren.
 
-Also ja - es macht Sinn, uns zu vertrauen, wenn wir sagen, wo 
+Also ja - es macht Sinn, uns zu vertrauen, wenn wir sagen, wo
 gerade Kontis sind.
 
 ## Was ist Mastodon und warum sollte ich es benutzen?
 
-Mastodon ist ein dezentrales soziales Netzwerk - so wie 
+Mastodon ist ein dezentrales soziales Netzwerk - so wie
 Twitter, nur ohne Monopol und Zentralismus.
-Ihr könnt Kurznachrichten (Toots) über alles mögliche 
+Ihr könnt Kurznachrichten (Toots) über alles mögliche
 schreiben, und euch mit anderen austauschen.
 
-Mastodon ist Open Source, Privatsphäre-freundlich und relativ 
+Mastodon ist Open Source, Privatsphäre-freundlich und relativ
 sicher vor Zensur.
 
-Um Mastodon zu benutzen, besucht diese Seite: 
+Um Mastodon zu benutzen, besucht diese Seite:
 [https://joinmastodon.org/](https://joinmastodon.org/)
         """
         mail_md = """# Immer up-to-date
@@ -429,15 +460,15 @@ zu schauen? Kein Problem. Unsere Mail Notifications benachrichtigen dich, wenn
 irgendwo Kontis gesehen werden.
 
 Wenn du uns deine E-Mail-Adresse gibst, kriegst du bei jedem Konti-Report eine
-Mail. Wenn du eine Mail-App auf dem Handy hast, so wie 
+Mail. Wenn du eine Mail-App auf dem Handy hast, so wie
 [K9Mail](https://k9mail.github.io/), kriegst du sogar eine Push Notification. So
 bist du immer Up-to-date über alles, was im Verkehrsnetz passiert.
 
 ## Keine Sorge
 
-Wir benutzen deine E-Mail-Adresse selbstverständlich für nichts anderes. Du 
+Wir benutzen deine E-Mail-Adresse selbstverständlich für nichts anderes. Du
 kannst die Benachrichtigungen jederzeit deaktivieren, mit jeder Mail wird ein
-unsubscribe-link mitgeschickt. 
+unsubscribe-link mitgeschickt.
         """
         db.execute("""INSERT INTO cities(user_id, city, markdown, mail_md,
                         masto_link, twit_link) VALUES(?,?,?,?,?,?)""",