Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
G
GitlabRSSSync
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Requirements
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Container registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
open-source
GitlabRSSSync
Commits
f703f832
Commit
f703f832
authored
6 years ago
by
Adam Harrison-Fuller
Browse files
Options
Downloads
Patches
Plain Diff
Make the retroactive creating of issues a config option
Signed-off-by:
Adam Harrison-Fuller
<
adam@adamhf.io
>
parent
91bbe2e1
Branches
Branches containing commit
Tags
Tags containing commit
No related merge requests found
Changes
4
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
README.md
+12
-14
12 additions, 14 deletions
README.md
config.yaml
+2
-1
2 additions, 1 deletion
config.yaml
config.yaml.example
+2
-0
2 additions, 0 deletions
config.yaml.example
main.go
+17
-7
17 additions, 7 deletions
main.go
with
33 additions
and
22 deletions
README.md
+
12
−
14
View file @
f703f832
...
@@ -7,11 +7,11 @@ our attention (Security Releases, Product Updates etc)
...
@@ -7,11 +7,11 @@ our attention (Security Releases, Product Updates etc)
## Avoiding Duplication
## Avoiding Duplication
We try to be as clever as is reasonably possible in terms of not duplicating RSS feed items into Gitlab.
We try to be as clever as is reasonably possible in terms of not duplicating RSS feed items into Gitlab.
A Redis
DB
is used to store the GUID/FeedID combination which is checked when assessing articles for synchronisation.
A Redis
database
is used to store the GUID/FeedID combination which is checked when assessing articles for synchronisation.
In addition we also add the RSS feed's item GUID at the bottom of the issue description. Before synchronising an RSS item
In addition we also add the RSS feed's item GUID at the bottom of the issue description. Before synchronising an RSS item
we run an issue search in the associated project, if we dont find the GUID in any issue we assume its not already been created.
we run an issue search in the associated project, if we dont find the GUID in any issue we assume its not already been created.
This helps to guard against scenarios where you lose the Redis
DB
and dont want RSS items reduplicating into Gitlab.
This helps to guard against scenarios where you lose the Redis
database
and dont want RSS items reduplicating into Gitlab.
If found in Gitlab it is marked as synchronised in the
local
database as well as printing an link to the existing issue(s) to stdout.
If found in Gitlab it is marked as synchronised in the
Redis
database as well as printing an link to the existing issue(s) to stdout.
## Limiting what is initially synced.
## Limiting what is initially synced.
Each feed entry in the config file can have an "added_since" property set. This is used to only sync RSS items that have a
Each feed entry in the config file can have an "added_since" property set. This is used to only sync RSS items that have a
...
@@ -42,14 +42,15 @@ feeds:
...
@@ -42,14 +42,15 @@ feeds:
| interval | int | yes | The interval in seconds between feed checks |
| interval | int | yes | The interval in seconds between feed checks |
### Feeds
### Feeds
| Attribute | Type | Required | Description |
| Attribute | Type | Required | Default | Description |
|-------------------|--------|----------|-------------------------------------------------------------------------------------------------------|
|-------------------|--------|----------|---------|----------------------------------------------------------------------------------------------------------|
| id | string | yes | A feed ID that is used internally for duplicate detection. |
| id | string | yes | n/a | A feed ID that is used internally for duplicate detection. |
| feed_url | string | yes | The URL of the feed |
| feed_url | string | yes | n/a | The URL of the feed |
| name | string | yes | A User friendly display name. |
| name | string | yes | n/a | A User friendly display name. |
| gitlab_project_id | int | yes | The Gitlab project ID to create issues under. |
| gitlab_project_id | int | yes | n/a | The Gitlab project ID to create issues under. |
| added_since | string | no | For longer RSS feeds specify a ISO 8601 DateTime to exclude items published/updated earlier than this |
| added_since | string | no | null | For longer RSS feeds specify a ISO 8601 DateTime to exclude items published/updated earlier than this |
| Labels | Array | no | A list of labels to add to created Issues |
| labels | Array | no | [] | A list of labels to add to created Issues |
| retroactive | bool | no | false | If true the issue in Gitlab will have the same creation time as the RSS feed items updates/published time|
...
@@ -75,8 +76,6 @@ docker run -e GITLAB_API_TOKEN=<INSERT_TOKEN> -e DATA_DIR=/data -e CONFIG_DIR=/a
...
@@ -75,8 +76,6 @@ docker run -e GITLAB_API_TOKEN=<INSERT_TOKEN> -e DATA_DIR=/data -e CONFIG_DIR=/a
docker-compose up
docker-compose up
```
```
## Prometheus Metrics
## Prometheus Metrics
Two metrics (above and beyond what are exposed by the Go Prometheus library) are exposed on :8080/metrics
Two metrics (above and beyond what are exposed by the Go Prometheus library) are exposed on :8080/metrics
*
last_run_time - The time of the last feed checks, useful for creating alerts to check for successful runs.
*
last_run_time - The time of the last feed checks, useful for creating alerts to check for successful runs.
...
@@ -90,6 +89,5 @@ Feed URL: https://cloud.google.com/feeds/kubernetes-engine-release-notes.xml
...
@@ -90,6 +89,5 @@ Feed URL: https://cloud.google.com/feeds/kubernetes-engine-release-notes.xml
Feed URL: https://cloud.google.com/feeds/kubernetes-engine-security-bulletins.xml
Feed URL: https://cloud.google.com/feeds/kubernetes-engine-security-bulletins.xml


## TODO
## TODO
*
Make the retroactive setting of the Gitlab creation time optional.
*
Make the retroactive setting of the Gitlab creation time optional.
\ No newline at end of file
This diff is collapsed.
Click to expand it.
config.yaml
+
2
−
1
View file @
f703f832
interval
:
30
interval
:
30
0
feeds
:
feeds
:
-
id
:
bbc_world
-
id
:
bbc_world
feed_url
:
http://feeds.bbci.co.uk/news/england/rss.xml
feed_url
:
http://feeds.bbci.co.uk/news/england/rss.xml
name
:
BBC World
name
:
BBC World
gitlab_project_id
:
11494338
gitlab_project_id
:
11494338
retroactive
:
true
labels
:
labels
:
-
BBC
-
BBC
-
id
:
gke_release_notes
-
id
:
gke_release_notes
...
...
This diff is collapsed.
Click to expand it.
config.yaml.example
+
2
−
0
View file @
f703f832
interval: 300
interval: 300
feeds:
feeds:
- id: reddit
- id: reddit
feed_url: https://www.reddit.com/.rss
feed_url: https://www.reddit.com/.rss
name: Reddit Front Page
name: Reddit Front Page
gitlab_project_id: 12345678
gitlab_project_id: 12345678
retroactive: false
labels:
labels:
- Reddit
- Reddit
This diff is collapsed.
Click to expand it.
main.go
+
17
−
7
View file @
f703f832
...
@@ -34,6 +34,7 @@ type Feed struct {
...
@@ -34,6 +34,7 @@ type Feed struct {
GitlabProjectID
int
`yaml:"gitlab_project_id"`
GitlabProjectID
int
`yaml:"gitlab_project_id"`
Labels
[]
string
Labels
[]
string
AddedSince
time
.
Time
`yaml:"added_since"`
AddedSince
time
.
Time
`yaml:"added_since"`
Retroactive
bool
}
}
type
EnvValues
struct
{
type
EnvValues
struct
{
...
@@ -93,17 +94,17 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
...
@@ -93,17 +94,17 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
log
.
Printf
(
"Checked feed: %s, New articles: %d, Old articles: %d"
,
feed
.
Name
,
len
(
newArticle
),
len
(
oldArticle
))
log
.
Printf
(
"Checked feed: %s, New articles: %d, Old articles: %d"
,
feed
.
Name
,
len
(
newArticle
),
len
(
oldArticle
))
for
_
,
item
:=
range
newArticle
{
for
_
,
item
:=
range
newArticle
{
var
t
ime
*
time
.
Time
var
itemT
ime
*
time
.
Time
// Prefer updated
t
ime to published
// Prefer updated
itemT
ime to published
if
item
.
UpdatedParsed
!=
nil
{
if
item
.
UpdatedParsed
!=
nil
{
t
ime
=
item
.
UpdatedParsed
itemT
ime
=
item
.
UpdatedParsed
}
else
{
}
else
{
t
ime
=
item
.
PublishedParsed
itemT
ime
=
item
.
PublishedParsed
}
}
if
t
ime
.
Before
(
feed
.
AddedSince
)
{
if
itemT
ime
.
Before
(
feed
.
AddedSince
)
{
log
.
Printf
(
"Ignoring '%s' as its date is before the specified AddedSince (Item: %s vs AddedSince: %s)
\n
"
,
log
.
Printf
(
"Ignoring '%s' as its date is before the specified AddedSince (Item: %s vs AddedSince: %s)
\n
"
,
item
.
Title
,
t
ime
,
feed
.
AddedSince
)
item
.
Title
,
itemT
ime
,
feed
.
AddedSince
)
redisClient
.
SAdd
(
feed
.
ID
,
item
.
GUID
)
redisClient
.
SAdd
(
feed
.
ID
,
item
.
GUID
)
continue
continue
}
}
...
@@ -123,11 +124,17 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
...
@@ -123,11 +124,17 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
body
=
item
.
Content
body
=
item
.
Content
}
}
now
:=
time
.
Now
()
issueTime
:=
&
now
if
feed
.
Retroactive
{
issueTime
=
itemTime
}
issueOptions
:=
&
gitlab
.
CreateIssueOptions
{
issueOptions
:=
&
gitlab
.
CreateIssueOptions
{
Title
:
gitlab
.
String
(
item
.
Title
),
Title
:
gitlab
.
String
(
item
.
Title
),
Description
:
gitlab
.
String
(
body
+
"
\n
"
+
item
.
GUID
),
Description
:
gitlab
.
String
(
body
+
"
\n
"
+
item
.
GUID
),
Labels
:
feed
.
Labels
,
Labels
:
feed
.
Labels
,
CreatedAt
:
t
ime
,
CreatedAt
:
issueT
ime
,
}
}
if
_
,
_
,
err
:=
gitlabClient
.
Issues
.
CreateIssue
(
feed
.
GitlabProjectID
,
issueOptions
);
err
!=
nil
{
if
_
,
_
,
err
:=
gitlabClient
.
Issues
.
CreateIssue
(
feed
.
GitlabProjectID
,
issueOptions
);
err
!=
nil
{
...
@@ -139,6 +146,9 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
...
@@ -139,6 +146,9 @@ func (feed Feed) checkFeed(redisClient *redis.Client, gitlabClient *gitlab.Clien
continue
continue
}
}
issuesCreatedCounter
.
Inc
()
issuesCreatedCounter
.
Inc
()
if
feed
.
Retroactive
{
log
.
Printf
(
"Retroactively issue setting date to %s"
,
itemTime
)
}
log
.
Printf
(
"Created Gitlab Issue '%s' in project: %d'
\n
"
,
item
.
Title
,
feed
.
GitlabProjectID
)
log
.
Printf
(
"Created Gitlab Issue '%s' in project: %d'
\n
"
,
item
.
Title
,
feed
.
GitlabProjectID
)
}
}
}
}
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment