Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.....X........................X..........................] duration: 545.909531ms FAIL (2025-07-27 01:38:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 545.944159ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.....X........................X.........................] duration: 427.502824ms FAIL (2025-07-27 01:38:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.541423ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 525.541312ms FAIL (2025-07-27 01:37:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.570069ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X.....X........................X........................] duration: 510.391877ms FAIL (2025-07-27 01:37:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.425765ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 7.27870842s FAIL (2025-07-27 01:37:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 7.278799592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 1.742980792s FAIL (2025-07-27 01:37:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.743012233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X.....X........................X.......................] duration: 525.071739ms FAIL (2025-07-27 01:37:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.106274ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 1.338099119s FAIL (2025-07-27 01:36:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.338135451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 659.311631ms FAIL (2025-07-27 01:36:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 659.428198ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X.....X........................X......................] duration: 434.603744ms FAIL (2025-07-27 01:36:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.658007ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXXX.....................................................] duration: 725.077764ms FAIL (2025-07-27 01:35:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 725.109563ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X..X......................................................] duration: 419.274664ms FAIL (2025-07-27 01:35:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.371072ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.X.....X........................X.....................] duration: 658.074625ms FAIL (2025-07-27 01:35:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 658.114466ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 961.16247ms FAIL (2025-07-27 01:35:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 961.191ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.14636477s FAIL (2025-07-27 01:35:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.146402615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..XXXXXX....................................................] duration: 1.09720109s FAIL (2025-07-27 01:34:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.097257025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.X..X.....................................................] duration: 459.569049ms FAIL (2025-07-27 01:34:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.77748ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.X.....X........................X....................] duration: 474.738241ms FAIL (2025-07-27 01:34:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.763009ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 228.0454ms FAIL (2025-07-27 01:34:14 UTC) orbiter warning: uplink: too many requests Duration: 228.078848ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXX.X..X....................................................] duration: 1.173278061s FAIL (2025-07-27 01:34:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.173313111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...XXXXXX...................................................] duration: 793.658178ms FAIL (2025-07-27 01:34:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 793.694135ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 936.155344ms FAIL (2025-07-27 01:33:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 936.192071ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 226.757949ms FAIL (2025-07-27 01:33:40 UTC) orbiter warning: uplink: too many requests Duration: 226.786126ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 580.507804ms FAIL (2025-07-27 01:33:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 580.533836ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXX.X..X...................................................] duration: 429.027388ms FAIL (2025-07-27 01:33:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.060703ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X......X.X.....X........................X...................] duration: 447.677302ms FAIL (2025-07-27 01:33:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.728487ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.210577881s FAIL (2025-07-27 01:33:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.210666839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 431.263688ms FAIL (2025-07-27 01:33:03 UTC) orbiter warning: uplink: too many requests Duration: 431.315194ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....XXXXXX..................................................] duration: 1.103891761s FAIL (2025-07-27 01:32:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.103942814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 720.052958ms FAIL (2025-07-27 01:32:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 720.07897ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 200.638534ms FAIL (2025-07-27 01:32:21 UTC) orbiter warning: uplink: too many requests Duration: 200.671211ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....XXXXXX.................................................] duration: 1.070917096s FAIL (2025-07-27 01:32:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.071020022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X......X.X.....X........................X..................] duration: 611.618598ms FAIL (2025-07-27 01:32:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 611.662701ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXX.X..X..................................................] duration: 457.132415ms FAIL (2025-07-27 01:32:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.162535ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 620.768075ms FAIL (2025-07-27 01:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 620.791801ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......XXXXXX................................................] duration: 622.906082ms FAIL (2025-07-27 01:31:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 622.943563ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXX.X..X.................................................] duration: 479.953879ms FAIL (2025-07-27 01:31:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.989283ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X......X.X.....X........................X.................] duration: 484.856602ms FAIL (2025-07-27 01:31:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 484.977528ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 450.213679ms FAIL (2025-07-27 01:31:14 UTC) orbiter warning: uplink: too many requests Duration: 450.250811ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 871.905624ms FAIL (2025-07-27 01:31:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 871.931513ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......XXXXXX...............................................] duration: 1.182985167s FAIL (2025-07-27 01:30:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.183024813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXXX.X..X................................................] duration: 455.089547ms FAIL (2025-07-27 01:30:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.22313ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 225.314163ms FAIL (2025-07-27 01:30:22 UTC) orbiter warning: uplink: too many requests Duration: 225.396326ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.X......X.X.....X........................X................] duration: 612.29996ms FAIL (2025-07-27 01:30:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 612.33978ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.022470353s FAIL (2025-07-27 01:30:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.022493701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........XXXXXX..............................................] duration: 2.787239459s FAIL (2025-07-27 01:29:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.787286741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 219.299864ms FAIL (2025-07-27 01:29:37 UTC) orbiter warning: uplink: too many requests Duration: 219.327217ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXXX.X..X...............................................] duration: 515.907059ms FAIL (2025-07-27 01:29:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.937041ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXXX.X..X..............................................] duration: 628.813541ms FAIL (2025-07-27 01:29:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 628.87139ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX.X......X.X.....X........................X...............] duration: 554.429273ms FAIL (2025-07-27 01:28:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 554.459079ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)