README.md 4.78 KB
Newer Older
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
1
# Gitlab RSS Sync
2
3
4
![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/adamhf/gitlabrsssync.svg)
![Docker Pulls](https://img.shields.io/docker/pulls/adamhf/gitlabrsssync.svg)

5
6
Create Gitlab issues from RSS Feeds with optional labelling.  Created to monitor RSS feeds and bring posts to
our attention (Security Releases, Product Updates etc)
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
7

8
9
## Avoiding Duplication
We try to be as clever as is reasonably possible in terms of not duplicating RSS feed items into Gitlab.
10
A Redis DB is used to store the GUID/FeedID combination which is checked when assessing articles for synchronisation.
11
12
In addition we also add the RSS feed's item GUID at the bottom of the issue description.  Before synchronising an RSS item
we run an issue search in the associated project, if we dont find the GUID in any issue we assume its not already been created.
13
This helps to guard against scenarios where you lose the Redis DB and dont want RSS items reduplicating into Gitlab.
Adam Harrison-Fuller's avatar
Tidyup    
Adam Harrison-Fuller committed
14
If found in Gitlab it is marked as synchronised in the local database as well as printing an link to the existing issue(s) to stdout.
15
16
17
18
19
20

## Limiting what is initially synced.
Each feed entry in the config file can have an "added_since" property set.  This is used to only sync RSS items that have a
Published/Updated date greater than the provided value.  This can be useful on RSS feeds where you dont want to import historic items,
just new posts going forward.

Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
21
## Config file
22
23
24
25
26

The config file **MUST** be named config.yaml, an example one is provided [here](config.yaml.example).  Below is a brief
 description of its contents.

```yaml
27
interval: 300
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
28
feeds:
29
30
31
32
33
34
  - id: test 
    feed_url: http://example.com/rss.xml
    name: Test Feed
    gitlab_project_id: 12345
    added_since: "2019-03-27T15:00:00Z"
    labels:
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
35
      - TestLabel
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
36
37
   - id: feed2
     ...
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
38
```
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
### Global
| Attribute | Type | Required | Description                                 |
|-----------|------|----------|---------------------------------------------|
| interval  | int  | yes      | The interval in seconds between feed checks |

### Feeds
| Attribute         | Type   | Required | Description                                                                                           |
|-------------------|--------|----------|-------------------------------------------------------------------------------------------------------|
| id                | string | yes      | A feed ID that is used internally for duplicate detection.                                            |
| feed_url          | string | yes      | The URL of the feed                                                                                   |
| name              | string | yes      | A User friendly display name.                                                                         |
| gitlab_project_id | int    | yes      | The Gitlab project ID to create issues under.                                                         |
| added_since       | string | no       | For longer RSS feeds specify a ISO 8601 DateTime to exclude items published/updated earlier than this |
| Labels            | Array  | no       | A list of labels to add to created Issues                                                             |


Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
55
56
57
58
59
60
61
62

## Docker
A Docker image is made available on [DockerHub](https://hub.docker.com/r/adamhf/gitlabrsssync)

### Required Environment Variables
* GITLAB_API_TOKEN - Gitlab personal access token that will be used to create Issues NOTE: You must have access to create
issues in the projects you specify in the config file.
* CONFIG_DIR - The directory the application should look for config.yaml in.
63
64
* REDIS_URL - The URL of the Redis host e.g. `redis:6379`
* REDIS_PASSWORD - Password for Redis, if an empty password is required set to `REDIS_PASSWORD=`
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
65
66

### Run it
67
68

#### Via Docker
69
```bash
70
docker run -e GITLAB_API_TOKEN=<INSERT_TOKEN> -e DATA_DIR=/data -e CONFIG_DIR=/app -v REDIS_URL=<REDIS_URL> -v REDIS_PASSWORD=<REDIS_PASSWORD> -v <PATH_TO_CONFIG_DIR>/config adamhf/rss-sync:latest
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
71
```
72
73
####Via docker-compose
```bash
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
74
75
docker-compose up
```
76
77


Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
78

79
80
81
82
83
## Prometheus Metrics
Two metrics (above and beyond what are exposed by the Go Prometheus library) are exposed on :8080/metrics
* last_run_time - The time of the last feed checks, useful for creating alerts to check for successful runs.
* issues_created - The total number of issues created in Gitlab, useful to check for runaways.

84
85
86
87
88
89
90
91
92
## Example Issues
### GKE Release Notes
Feed URL: https://cloud.google.com/feeds/kubernetes-engine-release-notes.xml
![GKE Release Notes](screenshots/GKEReleaseNotes.png "GKE Release Notes")
### GKE Security Updates
Feed URL: https://cloud.google.com/feeds/kubernetes-engine-security-bulletins.xml
![GKE Security updates](screenshots/GKESecurityUpdate.png "GKE Security updates")


Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
93
## TODO
Adam Harrison-Fuller's avatar
Adam Harrison-Fuller committed
94
* Make the retroactive setting of the Gitlab creation time optional.