query-1bc279cba0670fa7705f89436b5b7541
Discrepancies between new and old data: I've been looking at comparing the new data with what was already in place, prior to removing the old data. First of all, checking that all the items with old-style records (no term qualifier) now have a new-style record (term & date) as well. Gettinwikiwidit@ - I removed a few miscategorised state senators etc, and now there are just four, all of whom seem to be complicated "never took their seat" cases. corresponding new-style onenoPeople with old-style statements but All old-style statements where they have different end dates. There are 206 items with a discrepancy, as some are on both lists. They seem to fall into a few general groups. 149 cases where the old-style data and new-style data have different start dates, and 97 cases in the two gets a little more complicated. Most don't have any dates, but there are dataSo far, so good. Comparing the data is correct, and these may be missing from the source? Should hopefully be easy to add those short terms, though.existing are correct. I think in most of these the (Q2278283)Sheridan Downey is missing his extra month as a Senator, but the dates on his predecessor (Q9588)Richard Nixon was appointed to replace him that same day - so he served one day of the old term. The newly imported data treats Tsongas as going up until the end of his term, and then Kerry starting on the first day of the new term. In some cases there's a discrepancy on both items, on others just the one - eg (Q22316)John Kerry resigned as Senator one day early, and (Q283884)Paul Tsongas It's missing some very brief appointments. Lots of pre-1930s terms with 3 March vs 4 March end dates.01:33, 3 December 2020 (UTC)) talk (GettinwikiwiditMy inclination is to completely ignore the old model data. I'm not sure I follow the example mentioned above, but I'm happy to discuss how to handle any outlier cases. And finally a few are probably just typos on the original import from WP. just have a few days difference), presumably using different definitions,(Q44690)Lincoln Chafee and (Q76570207)Kelly Loeffler Other records have discrepancies over exact start/end dates (eg seems to treat older terms as running from March 4 to March 3 (one ends the day before the other starts), but since 1935 it lists them as January 3 to January 3 (starting and ending on the same day). Not sure if this is a legal change, or just one in convention. Senate chronological listFor the 3 March vs 4 March end dates, I'm not entirely sure which is correct. The 01:31, 3 December 2020 (UTC)) talk (GettinwikiwiditI was just about to write a note about this. For now I've left differences of a single day to be dealt with later. I'm guessing that some people assume you can't have two senators serving on the same day and are reluctant to record them as such where as others prefer having zero gap. I personally don't mind leaving this ambiguous because I think there's no way future consistency can be guaranteed. 00:22, 3 December 2020 (UTC)) talk (Andrew GrayI will try and find a bit of time over the next few days to fix up some of these discrepancies. It's probably worth mentioning that at this point only a handful of start and end times differ by more than a day from what can be found in the list of senators by state Wikipedia pages. I'll package up my software on post back so people can have a look at it. I'll also post back the remaining cases in case they can't be resolved trivially. 12:43, 3 December 2020 (UTC) 12:07, 3 December 2020 (UTC)) talk (GettinwikiwiditAlso per PC, I was planning on deleting all the old model claims by the end of this week. I'll hold off for another week or until I hear from you so as not to get in your way. Regards, 01:31, 3 December 2020 (UTC)) talk (GettinwikiwiditIt's probably worth mentioning that I've been making edits this morning. I'm not sure when you did your checks, but that seems like it's worth taking into account. 12:44, 3 December 2020 (UTC)) talk (Gettinwikiwidit: This sounds similar to the Replicating Wikipedia tables project mentioned above which I'm just finishing up. I'm reluctant to take the old Wikidata as authority, though and have been tending to use the Biographical Directory and Wikipedia as authoritative in that order, though there have been errors discovered in both. Andrew Gray@22:51, 8 December 2020 (UTC)) talk (Andrew Gray include the four "weird sort-of-Senators" who don't have new-style items with terms; I figured you'd want to keep those tagged for now so we don't lose track of them. notThis list does is the one I think is OK to delete; everyone on it has been checked with one of the date queries upthread, and is either resolved or a 3/4 March issue.This list: Got my wires crossed and replied on the project chat thread not this one, sorry :-). I think we're good to go assuming we're still going to do a later run to standardise on either 3 March or 4 March. I'm happy to line this up in the next day or two. Gettinwikiwidit@ 14:14, 8 December 2020 (UTC)) talk (Gettinwikiwidit: So are we good to remove the old model claims? Would you like to do it? I'm waiting on a clear go ahead. Andrew Gray@ 19:33, 7 December 2020 (UTC)) talk (Andrew Gray (as above), and the ones set to 3 March or 4 March. I've left these alone, as these seem to be ones where we'd be wanting to standardise term start/end dates anyway. (Q500046)Albert Gallatin And all end dates mismatches done, except for 19:03, 7 December 2020 (UTC)) talk (Andrew Gray (noted above, unclear how officially to model his "first term", 5-15 March 1849). Going to try and get the rest of the end dates done tonight. (Q923522)James Shields (who is the "not clear if he officially served as a Senator or not" case) and (Q500046)Albert Gallatin exceptAll start date mismatches now done 23:04, 4 December 2020 (UTC)) talk (Andrew Gray starting on 23/11/2002, while the chronological list has both on the same date (23rd). Which is probably more likely to be correct! Talent leaving on 25/11/2002 and CarnahanYeah, it's proving very helpful. I assume it's primarily drawn from the Biographical Dictionary, but there are a couple of discrepancies - eg the Dictionary 22:52, 4 December 2020 (UTC)) talk (Gettinwikiwidit looks great. Noting it here for posterity. This resource of yours. this edit: Thanks. Just saw Andrew Gray@ 23:55, 3 December 2020 (UTC)) talk (Andrew GraySounds good. I've done a bundle just now; where the old start/end date is incorrect, I've updated it so it matches the new data and thus will disappear from the queries to avoid having to look at it twice. Will keep working at it tomorrow. 22:40, 3 December 2020 (UTC)) talk (Gettinwikiwidit: I can wait. I've also saved off all P39 claims for senators as of today as a precaution. I'll just do this next week for the whole lot. Regards, Andrew Gray@ 22:34, 3 December 2020 (UTC)) talk (Andrew Gray show up on the above reports (about 90% of them), but if it's easier to wait and do them all in one fell swoop then that works for me. don'tIn terms of deletion, I think you should be good to remove all the old-style terms that of what I mean - Tsongas resigned after the election, and Kerry was appointed on 2 January, so he served for a single day of the 98th Congress and then started a new term in the 99th. These do seem to be real service, albeit something of a legal fiction (WP suggests it's a dodge to gain seniority), so probably we should include them? Hopefully they'll all pop out naturally as we work through these.this change is an exampleFor the "short terms" I mentioned above, : All sounds great. I'll chip away at some of these dates and if you're also coming at it state-by-state, it sounds like we'll get through them pretty fast! Agree entirely that the "old data" shouldn't be taken as authoritative, but if they disagree it looks like a good prompt to double-check for anomalies.Gettinwikiwidit@Noting here. We can clean up this edit later. This is where I left off:Start time off by 1 day or more:
Use at
- https://query.wikidata.org/sparql
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX wikibase: <http://wikiba.se/ontology#>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX bd: <http://www.bigdata.com/rdf#>
SELECT ?senLabel ?stmt ?prop (?oldStart AS ?lastChecked) ( ?newStart AS ?wikiLastChecked ) (?start AS ?current) WHERE {
VALUES (?sen ?seat ?term ?oldStart ?newStart) {
( wd:Q237220 wd:Q101498964 wd:Q4642363 "1925-01-08T00:00:00Z"^^xsd:dateTime "1924-12-17T00:00:00Z"^^xsd:dateTime )
( wd:Q271243 wd:Q101498968 wd:Q4642306 "1922-10-03T00:00:00Z"^^xsd:dateTime "1922-11-21T00:00:00Z"^^xsd:dateTime )
( wd:Q271023 wd:Q101498917 wd:Q4641029 "1907-01-29T00:00:00Z"^^xsd:dateTime "1907-01-22T00:00:00Z"^^xsd:dateTime )
( wd:Q368920 wd:Q101498920 wd:Q4646121 "1980-05-17T00:00:00Z"^^xsd:dateTime "1980-05-19T00:00:00Z"^^xsd:dateTime )
( wd:Q433351 wd:Q101498973 wd:Q2573610 "1797-12-08T00:00:00Z"^^xsd:dateTime "1797-12-11T00:00:00Z"^^xsd:dateTime )
( wd:Q1689229 wd:Q101498927 wd:Q2057259 "2002-11-23T00:00:00Z"^^xsd:dateTime "2002-11-25T00:00:00Z"^^xsd:dateTime )
( wd:Q343849 wd:Q101498941 wd:Q4643765 "1945-07-25T00:00:00Z"^^xsd:dateTime "1945-07-24T00:00:00Z"^^xsd:dateTime )
( wd:Q5997815 wd:Q101498867 wd:Q4645016 "1961-12-07T00:00:00Z"^^xsd:dateTime "1962-01-10T00:00:00Z"^^xsd:dateTime )
( wd:Q714960 wd:Q101498872 wd:Q2395126 "1800-04-03T00:00:00Z"^^xsd:dateTime "1800-05-03T00:00:00Z"^^xsd:dateTime )
( wd:Q457691 wd:Q101498984 wd:Q230796 "1796-11-09T00:00:00Z"^^xsd:dateTime "1796-12-08T00:00:00Z"^^xsd:dateTime )
( wd:Q1148970 wd:Q101498997 wd:Q4643047 "1928-04-04T00:00:00Z"^^xsd:dateTime "1928-04-05T00:00:00Z"^^xsd:dateTime )
( wd:Q1571326 wd:Q101498946 wd:Q4644951 "1960-03-16T00:00:00Z"^^xsd:dateTime "1960-03-23T00:00:00Z"^^xsd:dateTime )
( wd:Q1579558 wd:Q101499072 wd:Q4643468 "1938-02-01T00:00:00Z"^^xsd:dateTime "1938-02-11T00:00:00Z"^^xsd:dateTime )
( wd:Q323511 wd:Q101498874 wd:Q347346 "1991-05-08T00:00:00Z"^^xsd:dateTime "1991-05-09T00:00:00Z"^^xsd:dateTime )
( wd:Q925743 wd:Q101499000 wd:Q3556780 "1993-01-02T00:00:00Z"^^xsd:dateTime "1993-01-05T00:00:00Z"^^xsd:dateTime )
( wd:Q1840588 wd:Q101498950 wd:Q3556780 "1993-01-21T00:00:00Z"^^xsd:dateTime "1993-01-23T00:00:00Z"^^xsd:dateTime )
( wd:Q1000051 wd:Q101498961 wd:Q4643305 "1934-01-01T00:00:00Z"^^xsd:dateTime "1933-12-18T00:00:00Z"^^xsd:dateTime )
( wd:Q5120540 wd:Q101499054 wd:Q18740945 "2018-04-09T00:00:00Z"^^xsd:dateTime "2018-04-02T00:00:00Z"^^xsd:dateTime )
( wd:Q888132 wd:Q101498869 wd:Q168778 "2006-01-17T00:00:00Z"^^xsd:dateTime "2006-01-18T00:00:00Z"^^xsd:dateTime )
}
# VALUES ?sen { wd:Q1148970 }
?sen p:P39 ?stmt;
wdt:P31 wd:Q5.
?stmt ps:P39 wd:Q4416090;
pq:P768 ?seat;
pq:P2937 ?term;
pq:P580 ?start.
BIND("P580" AS ?prop)
FILTER( ?oldStart = ?start )
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}