Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 653.457644ms FAIL (2025-07-26 22:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 653.47974ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......................................XX..XXX..X..XX..X..XX] duration: 30.024568075s FAIL (2025-07-26 22:08:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:60284->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857365078 (15m50.638298459s) [seg_pos={0 0}, upload=241]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 22:07:34.881995175 (15m50.662928543s) [upload=241]: Done uploading segments... Jul 26 22:07:34.882039661 (15m50.662973037s) [upload=241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 22:07:34.987498076 (15m50.768431441s) [upload=250]: Starting upload Jul 26 22:07:34.987564063 (15m50.768497438s) [upload=250]: Done starting upload Jul 26 22:07:34.987689434 (15m50.768622805s) [upload=250]: Uploading segments... Jul 26 22:07:34.987824291 (15m50.768757667s) [upload=250]: Got next segment. Inline: false Jul 26 22:07:34.987885176 (15m50.768818553s) [upload=250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:07:34.987891830 (15m50.768825202s) [upload=250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:07:35.016826140 (15m50.797759515s) [seg_pos={0 0}, upload=250]: Begin upload segment... Jul 26 22:07:35.016921473 (15m50.797854851s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017468447 (15m50.798401832s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017875249 (15m50.798808635s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018145431 (15m50.799078811s) [seg_pos={0 0}, upload=250]: Done begin upload segment. Jul 26 22:07:35.018172581 (15m50.799105965s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018586538 (15m50.799519922s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018978715 (15m50.799912097s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019607879 (15m50.800541265s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019859169 (15m50.800792548s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020584667 (15m50.801518035s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020634450 (15m50.801567817s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021358305 (15m50.802291683s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021726279 (15m50.802659658s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023351032 (15m50.804284442s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023838014 (15m50.8047714s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024344842 (15m50.805278227s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024738414 (15m50.805671796s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.112546258 (15m50.893479637s) [upload=250]: Next returned nil segment Jul 26 22:07:35.112579119 (15m50.8935125s) [upload=250]: Waiting for error group managing segments... Jul 26 22:07:35.115448922 (15m50.896382284s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.115834265 (15m50.896767627s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116276376 (15m50.897209752s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116721260 (15m50.897654636s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117075039 (15m50.898008407s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117447964 (15m50.898381334s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117922945 (15m50.898856313s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118400510 (15m50.899333892s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118978228 (15m50.899911603s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.124493998 (15m50.905427362s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.569833987 (15m51.350767353s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:08:04.957316577 (16m20.738249955s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 159.223.143.39:42898->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957422534 (16m20.73835593s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:33464->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957456634 (16m20.738390015s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:60924->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957485814 (16m20.738419177s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:43946->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957499116 (16m20.738432493s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:40640->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957544370 (16m20.738477756s) [seg_pos={0 0}, upload=250]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:08:04.979268646 (16m20.760202015s) [upload=250]: Done uploading segments... Jul 26 22:08:04.979309377 (16m20.760242755s) [upload=250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.024590963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................X..X........XXX.XXX.] duration: 588.009088ms FAIL (2025-07-26 22:07:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 588.036895ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 565.482094ms FAIL (2025-07-26 22:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 565.516576ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.041606098s FAIL (2025-07-26 22:07:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== , piece_id=BCZOBDMVEVOQAOGUS566GMMWHO4G2YRGPKVFVJLLNN3UWK644Q5Q, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877006092 (14m56.65793947s) [noise=true , address=104.36.82.41:4783 , piece_id=EAYWWTXHBEUQWK2MTOASIZJ7WUZYHHO3NNJA72OLUJ357VWTRV3Q, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877310238 (14m56.658243613s) [noise=true , address=104.36.82.52:4092 , piece_id=Y6JU6PBUAHAOWHFYPEN2Q3OTY6YEM4QPMDLK3WXFY6QLSE4GHEMA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877379938 (14m56.658313324s) [noise=true , address=104.36.82.46:4508 , piece_id=Q6CPIWSJS67CFDO3BQINFHIATZHAE5UA632KP3BDML6QVMDQAFXQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877637090 (14m56.65857047s) [noise=true , address=77.90.31.21:4882 , piece_id=JAODWQCKYYGIGJTWLBFLZ54SARW3RPM34OWHC2VBUHCL24OIHZWQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877819176 (14m56.658752541s) [noise=true , address=104.36.82.53:4150 , piece_id=ZEEGY53KJBANKX75IL3IHXSGGWT4VNPWPK6WQAIFZMM76GP3IZ7A, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877948680 (14m56.658882057s) [noise=true , address=79.127.201.215:4358 , piece_id=F5FPDFY525VHMUREAJWTMVWM4GEH4LUCWVVJQHF3M766AAORYMVQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.878274977 (14m56.659208357s) [noise=true , address=104.36.82.15:4283 , piece_id=463BFK2XQGX4RYFKUUZWA47NB4QTPIY3XU2E2MAO5KYBWCRP6IFQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.878596455 (14m56.659529834s) [noise=true , address=104.36.82.35:4602 , piece_id=VTQQJNCGPO6XIZGHVTHAKREY7M7D55URTJX4JAKDXXHLUHWZU26A, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.879263348 (14m56.660196733s) [noise=true , address=104.36.82.56:4261 , piece_id=EFIA3MY2I7H2HFXUMW4YCPV65SV5IJ6OG2GJEUW742G2N4W243QQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.879823357 (14m56.660756719s) [noise=true , address=104.36.82.10:4098 , piece_id=CXVIW4YYSBS3LARQVQ3AXWBVRCRZ7X7LP5GZSR3KIC2OOS4POZXA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.880462649 (14m56.661396034s) [noise=true , address=64.49.13.155:4916 , piece_id=2FKOFXTHMCXDQVGEJ3MM7U72VLSSMYW6FRDBJE3F2XZCKYFPS3ZA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.942586552 (14m56.723519942s) [upload=234]: Next returned nil segment Jul 26 22:06:40.942607586 (14m56.723540972s) [upload=234]: Waiting for error group managing segments... Jul 26 22:06:40.951178134 (14m56.732111514s) [noise=true , address=77.90.31.21:4882 , piece_id=JAODWQCKYYGIGJTWLBFLZ54SARW3RPM34OWHC2VBUHCL24OIHZWQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.480824114 (14m57.261757476s) [noise=true , address=79.127.205.234:4613 , piece_id=25SSVYOEJI2PVKEB4NTP2PILDOWUAMZL7I2QR7UFNKYLEU3JIMHA, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.566016795 (14m57.346950175s) [noise=true , address=64.49.13.155:4916 , piece_id=2FKOFXTHMCXDQVGEJ3MM7U72VLSSMYW6FRDBJE3F2XZCKYFPS3ZA, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.865601601 (14m57.646534984s) [noise=true , address=156.146.43.225:4448 , piece_id=ILJ54PWU4YEYQ6DIBEKPJUEY3RS6EPBKXFLYGFCLESRYBFBPJKCQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:06:43.354758170 (14m59.135691615s) [noise=true , address=79.127.201.215:4358 , piece_id=F5FPDFY525VHMUREAJWTMVWM4GEH4LUCWVVJQHF3M766AAORYMVQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:07:40.800116216 (15m56.581049577s) [noise=true , address=104.36.82.53:4150 , piece_id=ZEEGY53KJBANKX75IL3IHXSGGWT4VNPWPK6WQAIFZMM76GP3IZ7A, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800316561 (15m56.581249949s) [noise=true , address=104.36.82.56:4261 , piece_id=EFIA3MY2I7H2HFXUMW4YCPV65SV5IJ6OG2GJEUW742G2N4W243QQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: write tcp 159.223.143.39:38844->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800383993 (15m56.581317369s) [noise=true , address=104.36.82.10:4098 , piece_id=CXVIW4YYSBS3LARQVQ3AXWBVRCRZ7X7LP5GZSR3KIC2OOS4POZXA, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: rpc: dial tcp 104.36.82.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800464580 (15m56.581397956s) [noise=true , address=104.36.82.35:4602 , piece_id=VTQQJNCGPO6XIZGHVTHAKREY7M7D55URTJX4JAKDXXHLUHWZU26A, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12oPZgRG4QqgnQLo1fgyGYoX1zd34wsd5n7p7L3mRbPD8B4JUuB, address:104.36.82.35:4602): protocol: write tcp 159.223.143.39:47488->104.36.82.35:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800520570 (15m56.581453946s) [noise=true , address=104.36.82.36:4640 , piece_id=K7HNZOSXETXS2Y77ICWIXX2BG37L4Y6QRZKJIJLGRHCZ6RKT3XHQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1cL33G9zW6AsBkSfbrYC5DH1hk3ffBoNnt6CoeEy1D8SKS2hhS, address:104.36.82.36:4640): protocol: write tcp 159.223.143.39:54832->104.36.82.36:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800559320 (15m56.581492695s) [noise=true , address=104.36.82.47:4525 , piece_id=LCXBDWBOFL7GTBNVUZA6K7TREVIR6INMVXN5XYZQQGPN67TBLM5Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:37796->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800599019 (15m56.58153238s) [noise=true , address=104.36.82.41:4783 , piece_id=EAYWWTXHBEUQWK2MTOASIZJ7WUZYHHO3NNJA72OLUJ357VWTRV3Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 159.223.143.39:36450->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800636207 (15m56.581569582s) [noise=true , address=104.36.82.52:4092 , piece_id=Y6JU6PBUAHAOWHFYPEN2Q3OTY6YEM4QPMDLK3WXFY6QLSE4GHEMA, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:43048->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800676301 (15m56.581609676s) [noise=true , address=104.36.82.46:4508 , piece_id=Q6CPIWSJS67CFDO3BQINFHIATZHAE5UA632KP3BDML6QVMDQAFXQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800719140 (15m56.581652501s) [noise=true , address=104.36.82.15:4283 , piece_id=463BFK2XQGX4RYFKUUZWA47NB4QTPIY3XU2E2MAO5KYBWCRP6IFQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU, address:104.36.82.15:4283): protocol: write tcp 159.223.143.39:34268->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800739495 (15m56.581672857s) [noise=true , address=104.36.82.40:4950 , piece_id=S52SGC3KQSNK7GDRBVYU3YKXKF6QFKWYF6CGT4IYGHATBHSMH33Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800757982 (15m56.581691357s) [noise=true , address=104.36.82.54:4189 , piece_id=ZB6LBJCD2OBUW63NIQ3JDFOTGHTOWQBWMW2VCECEXIVM26HWJYYQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800895980 (15m56.581829358s) [noise=true , address=104.36.82.38:4707 , piece_id=BCZOBDMVEVOQAOGUS566GMMWHO4G2YRGPKVFVJLLNN3UWK644Q5Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:104.36.82.38:4707): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:07:40.800911905 (15m56.581845281s) [seg_pos={0 0}, upload=234]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 22:07:40.839274557 (15m56.620207933s) [upload=234]: Done uploading segments... Jul 26 22:07:40.839304866 (15m56.620238243s) [upload=234]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.041622514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XX..XXX..X..XX..X..X] duration: 30.027344199s FAIL (2025-07-26 22:07:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ce. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: context canceled; context canceled Jul 26 22:06:31.233378398 (14m47.014311764s) [seg_pos={0 0}, upload=232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:06:31.233469029 (14m47.014402405s) [upload=232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:06:31.233476080 (14m47.014409455s) [upload=232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:06:31.278182361 (14m47.059115727s) [upload=232]: Done uploading segments... Jul 26 22:06:31.278200919 (14m47.059134284s) [upload=232]: Upload finished. err: <nil> Jul 26 22:07:04.886004988 (15m20.666938357s) [upload=241]: Starting upload Jul 26 22:07:04.886083010 (15m20.667016377s) [upload=241]: Done starting upload Jul 26 22:07:04.886216997 (15m20.667150366s) [upload=241]: Uploading segments... Jul 26 22:07:04.886399538 (15m20.667332914s) [upload=241]: Got next segment. Inline: false Jul 26 22:07:04.886437605 (15m20.66737098s) [upload=241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:07:04.886442975 (15m20.66737635s) [upload=241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:07:04.913653381 (15m20.694586766s) [seg_pos={0 0}, upload=241]: Begin upload segment... Jul 26 22:07:04.913704315 (15m20.694637701s) [seg_pos={0 0}, upload=241]: Done begin upload segment. Jul 26 22:07:04.913734987 (15m20.694668373s) [noise=true , address=77.90.31.16:4735 , piece_id=ABL77ZP3AYR55JMLZSJQPNS5MADKSVXLRG4HNNHCQ3RI6XJVSN3A, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.914292152 (15m20.69522554s) [noise=true , address=104.36.82.6:4873 , piece_id=RMOJ6XAZROKNZ5UXQNHZQSK72CLQDF4YZC7YPF6XO5LQBHKSLWYA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.914734764 (15m20.69566815s) [noise=true , address=45.91.116.7:4011 , piece_id=32PGHV3RKESO3I7NHKAT6FPBLUPA2YXX7YBMIVDRZB3T3CZMQSIQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.915201856 (15m20.696135238s) [noise=true , address=104.36.82.8:4971 , piece_id=JTAWVK5TNVLYXEAH3I76YS6LEMBWB5VXXZIVSWVJCP2MTK62CNBQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.915643883 (15m20.69657727s) [noise=true , address=45.91.116.11:4117 , piece_id=S4C6HHEXO6THUQKOT4KDCCUAXZJROY4O4E5TBBBGUY4G5CL35ERA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.916071795 (15m20.697005181s) [noise=true , address=45.91.116.12:4225 , piece_id=HO33CJOKMCLPHCQBMBN55GYORDG2LBZKJSZU75FP7VO7DJWLO74Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.916615660 (15m20.697549043s) [noise=true , address=45.91.116.12:4226 , piece_id=7JZIBEFPYFEZHAJJBTJ6HZWJKCQ7HJ4RNTAYZ3HSWOM2NSWAC6QA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917030219 (15m20.697963595s) [noise=true , address=45.91.116.10:4082 , piece_id=KSATKOX4SOOPYQNC6VE3QS3OZODW3HVQAJJMA2UIXIOQXSZI7YAQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917308642 (15m20.698242017s) [noise=true , address=77.90.31.19:4841 , piece_id=5HRMIX7NQ7TCRXMYXITOUXTYM242KKI4IC2LJXWHL4ZYI5EMRUVA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917581133 (15m20.698514508s) [noise=true , address=45.91.116.5:4297 , piece_id=PAQ47YOYSS7A4WZVWFVUVZJQ3VROOJNTYKLATHIUC2X4UKTA4FGQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917845509 (15m20.698778884s) [noise=true , address=45.91.116.14:4191 , piece_id=MUN6GXT527TPRWO7R4STIPA4W7F2JA5CQGYZ4MB6Z5IMMPZ4E7WA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.918196794 (15m20.699130178s) [noise=true , address=104.36.82.3:4765 , piece_id=MPM2QMWXZAXYLOBESY4A5QKMSVMABKVEJQLT3ZY7FHBN6JD5OW6Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.918601274 (15m20.699534659s) [noise=true , address=45.91.116.13:4153 , piece_id=RXYGO2GR3Y7EWO2Z7CVFWLXGUVTPRVBM7YNIRFUCVLGOLYHSRZRA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.921774832 (15m20.702708217s) [noise=true , address=104.36.82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.922268298 (15m20.703201683s) [noise=true , address=77.90.31.10:4517 , piece_id=QEPL5S5FEYYPUTKHWJYZPSGIQXSD2J3ISA64KGDBS5HYEYJRWZKQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.922742454 (15m20.70367584s) [noise=true , address=104.36.82.7:4910 , piece_id=QCAXQDKO44Q22NJ5SCEX44Q2QEUD4CQDPOWV6MHAIWM4PI4F6SAQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:05.008625277 (15m20.789558673s) [upload=241]: Next returned nil segment Jul 26 22:07:05.008658994 (15m20.789592389s) [upload=241]: Waiting for error group managing segments... Jul 26 22:07:05.012782416 (15m20.7937158s) [noise=true , address=45.91.116.12:4226 , piece_id=7JZIBEFPYFEZHAJJBTJ6HZWJKCQ7HJ4RNTAYZ3HSWOM2NSWAC6QA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013480505 (15m20.794413887s) [noise=true , address=45.91.116.5:4297 , piece_id=PAQ47YOYSS7A4WZVWFVUVZJQ3VROOJNTYKLATHIUC2X4UKTA4FGQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013559337 (15m20.794492712s) [noise=true , address=45.91.116.7:4011 , piece_id=32PGHV3RKESO3I7NHKAT6FPBLUPA2YXX7YBMIVDRZB3T3CZMQSIQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013876421 (15m20.794809783s) [noise=true , address=45.91.116.13:4153 , piece_id=RXYGO2GR3Y7EWO2Z7CVFWLXGUVTPRVBM7YNIRFUCVLGOLYHSRZRA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.014266643 (15m20.795200008s) [noise=true , address=45.91.116.14:4191 , piece_id=MUN6GXT527TPRWO7R4STIPA4W7F2JA5CQGYZ4MB6Z5IMMPZ4E7WA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.014367475 (15m20.795300845s) [noise=true , address=45.91.116.12:4225 , piece_id=HO33CJOKMCLPHCQBMBN55GYORDG2LBZKJSZU75FP7VO7DJWLO74Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.015368047 (15m20.796301413s) [noise=true , address=45.91.116.11:4117 , piece_id=S4C6HHEXO6THUQKOT4KDCCUAXZJROY4O4E5TBBBGUY4G5CL35ERA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.016610390 (15m20.79754376s) [noise=true , address=77.90.31.19:4841 , piece_id=5HRMIX7NQ7TCRXMYXITOUXTYM242KKI4IC2LJXWHL4ZYI5EMRUVA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.023964727 (15m20.804898094s) [noise=true , address=45.91.116.10:4082 , piece_id=KSATKOX4SOOPYQNC6VE3QS3OZODW3HVQAJJMA2UIXIOQXSZI7YAQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.768989621 (15m21.54992299s) [noise=true , address=77.90.31.16:4735 , piece_id=ABL77ZP3AYR55JMLZSJQPNS5MADKSVXLRG4HNNHCQ3RI6XJVSN3A, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:34.857092253 (15m50.638025619s) [noise=true , address=104.36.82.7:4910 , piece_id=QCAXQDKO44Q22NJ5SCEX44Q2QEUD4CQDPOWV6MHAIWM4PI4F6SAQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857153354 (15m50.638086721s) [noise=true , address=104.36.82.6:4873 , piece_id=RMOJ6XAZROKNZ5UXQNHZQSK72CLQDF4YZC7YPF6XO5LQBHKSLWYA, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857217544 (15m50.638150909s) [noise=true , address=104.36.82.3:4765 , piece_id=MPM2QMWXZAXYLOBESY4A5QKMSVMABKVEJQLT3ZY7FHBN6JD5OW6Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857273731 (15m50.638207106s) [noise=true , address=77.90.31.10:4517 , piece_id=QEPL5S5FEYYPUTKHWJYZPSGIQXSD2J3ISA64KGDBS5HYEYJRWZKQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857280934 (15m50.638214302s) [noise=true , address=104.36.82.8:4971 , piece_id=JTAWVK5TNVLYXEAH3I76YS6LEMBWB5VXXZIVSWVJCP2MTK62CNBQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857334028 (15m50.638267392s) [noise=true , address=104.36.82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:60284->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857365078 (15m50.638298459s) [seg_pos={0 0}, upload=241]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 22:07:34.881995175 (15m50.662928543s) [upload=241]: Done uploading segments... Jul 26 22:07:34.882039661 (15m50.662973037s) [upload=241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.027385728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................X..X........XXX.XXX] duration: 30.038421607s FAIL (2025-07-26 22:07:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.038442419s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.08825315s FAIL (2025-07-26 22:07:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.088574587s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X..X........XXX.XX] duration: 30.033474142s FAIL (2025-07-26 22:06:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.033498718s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001445943s FAIL (2025-07-26 22:06:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001469192s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.025673188s FAIL (2025-07-26 22:06:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== os={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.756773054 (13m56.537706439s) [noise=true , address=104.36.82.7:4926 , piece_id=KY7HLSG3GGWWYPXBNSOYCGCSZ7Y7LB2ZB574G7DIVRLQ23ZWYJOQ, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.759455929 (13m56.540389315s) [noise=true , address=104.36.82.54:4162 , piece_id=OEVQPVY2NCCKZUMYQDWNYDJCHSHSWMT3WBEJIBEGUVUVRYQDS2WA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.759956023 (13m56.540889399s) [noise=true , address=104.36.82.2:4755 , piece_id=J5WJ7FQ4K4WFBBHIW4ZXJUOX3ML6WCI5YOS5BTY5YCBQ2RTUCPWA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.760512420 (13m56.541445805s) [noise=true , address=104.36.82.53:4132 , piece_id=6ZTG5RP36ICMFHU5QM5O6AAG4NJXTZHKCYDEMJD6R3QNXPS5I3ZA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.760997556 (13m56.54193094s) [noise=true , address=79.127.201.215:4348 , piece_id=BGIT6ALO5WXNURCZT3VJOBSSKQLP4R46T4EQEFPJXVJFYKLJRPNA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.761476982 (13m56.542410367s) [noise=true , address=79.127.219.47:4061 , piece_id=IG5CZCTNJJGMAH6NHV4MFNLPJCS5MDNX7UZP63FWX5OKLSZNF5WA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.761900446 (13m56.542833821s) [noise=true , address=104.36.82.52:4087 , piece_id=UKFGL72QTZPWNFAQOV232AFFPAU4JALLU7RHUNXAOCIXRUMI7SMQ, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.762273618 (13m56.543207003s) [noise=true , address=64.49.13.138:4482 , piece_id=NIJ77KAAMLZK7XNGZTGO2FAUPDGLUL2QYKEPA5DCE3OOB4HJUPQA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.762712987 (13m56.543646373s) [noise=true , address=104.36.82.47:4537 , piece_id=6QTRBWL5LYHLXVTYN64ZXRZIBHUALS2Q5VZ3IGYZAFSH4BSTPYKQ, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.763226258 (13m56.544159639s) [noise=true , address=104.36.82.36:4649 , piece_id=F7CXXPWXTFTCD7NYVVMRF7WRH5GT73Q2RNCDP2WQEB4EMVMVXBHA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.763718181 (13m56.54465156s) [noise=true , address=104.36.82.18:4393 , piece_id=ER6TMZ5SPX3EBM7ZHR7GKRXKRFQTSWC3OPT24D6J4ISMVMNFPDSA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.764221457 (13m56.545154842s) [noise=true , address=104.36.82.42:4818 , piece_id=XMOF55F3KKRVGQNVCUSOUJGB627FYFGVWY4GFFIVUKPTSOMAAXZA, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.764728444 (13m56.545661824s) [noise=true , address=104.36.82.41:4795 , piece_id=OXMFGHHFU7U6RY52RHUPQXZDT5LDXPB7DB2VLJ6GN4SJZR7IA4RQ, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.765200971 (13m56.546134353s) [noise=true , address=104.36.82.3:4798 , piece_id=6PM6J5EKVHSIPQVA6U67YZL3FCJZAF5EOXCGUNL2VEJO7SUFXSBQ, seg_pos={0 0}, upload=217]: Uploading piece... Jul 26 22:05:40.832595387 (13m56.61352875s) [upload=217]: Next returned nil segment Jul 26 22:05:40.832616707 (13m56.61355009s) [upload=217]: Waiting for error group managing segments... Jul 26 22:05:40.838003611 (13m56.618937001s) [noise=true , address=79.127.219.38:4005 , piece_id=GK3EDONUQG6N4HUEO65KQ55GJNLRLSHT2YMTZ5AJNASMSEDIJTAA, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:05:40.864960525 (13m56.645893895s) [noise=true , address=104.36.82.7:4926 , piece_id=KY7HLSG3GGWWYPXBNSOYCGCSZ7Y7LB2ZB574G7DIVRLQ23ZWYJOQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:05:45.791559120 (14m1.572492505s) [noise=true , address=79.127.219.47:4061 , piece_id=IG5CZCTNJJGMAH6NHV4MFNLPJCS5MDNX7UZP63FWX5OKLSZNF5WA, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:05:47.576560911 (14m3.357494292s) [noise=true , address=64.49.13.138:4482 , piece_id=NIJ77KAAMLZK7XNGZTGO2FAUPDGLUL2QYKEPA5DCE3OOB4HJUPQA, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:05:48.718450103 (14m4.499383539s) [noise=true , address=79.127.201.215:4348 , piece_id=BGIT6ALO5WXNURCZT3VJOBSSKQLP4R46T4EQEFPJXVJFYKLJRPNA, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:05:50.977821125 (14m6.758754505s) [noise=true , address=64.49.13.140:4508 , piece_id=TUTRW4TUZGG6Q7DATMZRCM2WDJE7HH3JWKUBL2FWBWUH7NRG2JVQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: <nil> Jul 26 22:06:40.688300257 (14m56.469233632s) [noise=true , address=104.36.82.52:4087 , piece_id=UKFGL72QTZPWNFAQOV232AFFPAU4JALLU7RHUNXAOCIXRUMI7SMQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688363593 (14m56.469296967s) [noise=true , address=104.36.82.47:4537 , piece_id=6QTRBWL5LYHLXVTYN64ZXRZIBHUALS2Q5VZ3IGYZAFSH4BSTPYKQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688418683 (14m56.469352058s) [noise=true , address=104.36.82.46:4502 , piece_id=KLR6H4GIPMOXPTBAXR6EJ3UO5TB2T3KK7M54QEVJUHKM25JNJ24A, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688473439 (14m56.469406814s) [noise=true , address=104.36.82.18:4393 , piece_id=ER6TMZ5SPX3EBM7ZHR7GKRXKRFQTSWC3OPT24D6J4ISMVMNFPDSA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12fDTUq74fpQVaFNsfxPmKwMpG9eUfFDcjPd5FutuqXiBp9R5EN, address:104.36.82.18:4393): protocol: write tcp 159.223.143.39:47104->104.36.82.18:4393: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688589510 (14m56.469522886s) [noise=true , address=104.36.82.55:4212 , piece_id=DPA5RCPNTJDTTHGNSYPD2KAFKXWN7JAP4OJVZSIRUW6WD5D34PPA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688670406 (14m56.469603768s) [noise=true , address=104.36.82.54:4162 , piece_id=OEVQPVY2NCCKZUMYQDWNYDJCHSHSWMT3WBEJIBEGUVUVRYQDS2WA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: rpc: dial tcp 104.36.82.54:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688742833 (14m56.469676218s) [noise=true , address=104.36.82.2:4755 , piece_id=J5WJ7FQ4K4WFBBHIW4ZXJUOX3ML6WCI5YOS5BTY5YCBQ2RTUCPWA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: rpc: dial tcp 104.36.82.2:4755: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688769109 (14m56.469702484s) [noise=true , address=104.36.82.41:4795 , piece_id=OXMFGHHFU7U6RY52RHUPQXZDT5LDXPB7DB2VLJ6GN4SJZR7IA4RQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12ovwD5mScxxBf7RtRjAbBqRiqgAmTC8ajN3oBAXSaWrDQ1t9xk, address:104.36.82.41:4795): protocol: rpc: dial tcp 104.36.82.41:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688810128 (14m56.469743504s) [noise=true , address=104.36.82.36:4649 , piece_id=F7CXXPWXTFTCD7NYVVMRF7WRH5GT73Q2RNCDP2WQEB4EMVMVXBHA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: rpc: dial tcp 104.36.82.36:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688823947 (14m56.469757322s) [noise=true , address=104.36.82.3:4798 , piece_id=6PM6J5EKVHSIPQVA6U67YZL3FCJZAF5EOXCGUNL2VEJO7SUFXSBQ, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12JeRiSx1G5wZ32joVCBAffXaKPpuvLKDr2MrfqCTA1B5VM7WDg, address:104.36.82.3:4798): protocol: rpc: dial tcp 104.36.82.3:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688862815 (14m56.469796194s) [noise=true , address=104.36.82.42:4818 , piece_id=XMOF55F3KKRVGQNVCUSOUJGB627FYFGVWY4GFFIVUKPTSOMAAXZA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: rpc: dial tcp 104.36.82.42:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688892655 (14m56.46982603s) [noise=true , address=104.36.82.53:4132 , piece_id=6ZTG5RP36ICMFHU5QM5O6AAG4NJXTZHKCYDEMJD6R3QNXPS5I3ZA, seg_pos={0 0}, upload=217]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: write tcp 159.223.143.39:47290->104.36.82.53:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:06:40.688910827 (14m56.469844193s) [seg_pos={0 0}, upload=217]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:06:40.711993386 (14m56.492926762s) [upload=217]: Done uploading segments... Jul 26 22:06:40.712025276 (14m56.492958658s) [upload=217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.025697807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................XX..XXX..X..XX..X..] duration: 1.042012161s FAIL (2025-07-26 22:06:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 26 22:04:39.324461853 (12m55.105395215s) [noise=true , address=104.36.82.14:4223 , piece_id=IMO2TU5HFL3V7DBP4L6J7FNXKNLGG4S2CF5S5FFYRH6N5GP6NRPQ, seg_pos={0 0}, upload=202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 159.223.143.39:47066->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:39.324475303 (12m55.105408689s) [seg_pos={0 0}, upload=202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:04:39.324563844 (12m55.105497224s) [upload=202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:04:39.324568956 (12m55.105502339s) [upload=202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:04:39.462300657 (12m55.243234039s) [upload=202]: Done uploading segments... Jul 26 22:04:39.462336869 (12m55.243270249s) [upload=202]: Upload finished. err: <nil> Jul 26 22:04:59.791460713 (13m15.572394091s) [upload=208]: Starting upload Jul 26 22:04:59.791550369 (13m15.572483756s) [upload=208]: Done starting upload Jul 26 22:04:59.791719355 (13m15.572652723s) [upload=208]: Uploading segments... Jul 26 22:04:59.791892875 (13m15.572826261s) [upload=208]: Got next segment. Inline: false Jul 26 22:04:59.791950644 (13m15.572884027s) [upload=208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:04:59.791959841 (13m15.572893221s) [upload=208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:04:59.820721780 (13m15.601655158s) [seg_pos={0 0}, upload=208]: Begin upload segment... Jul 26 22:04:59.820785657 (13m15.601719027s) [seg_pos={0 0}, upload=208]: Done begin upload segment. Jul 26 22:04:59.820827367 (13m15.601760746s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821370184 (13m15.602303679s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821775197 (13m15.602708574s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822140231 (13m15.603073612s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822522075 (13m15.603455452s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822895726 (13m15.603829106s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828346851 (13m15.609280241s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828835238 (13m15.609768619s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829217860 (13m15.610151245s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829597682 (13m15.610531063s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830018233 (13m15.610951615s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830405152 (13m15.611338532s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830780717 (13m15.611714095s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831187997 (13m15.612121374s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831568937 (13m15.612502319s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831971236 (13m15.612904619s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.941095443 (13m15.722028823s) [upload=208]: Next returned nil segment Jul 26 22:04:59.941117988 (13m15.722051374s) [upload=208]: Waiting for error group managing segments... Jul 26 22:04:59.952061597 (13m15.732994966s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.952764229 (13m15.733697607s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953403179 (13m15.734336555s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953649158 (13m15.734582524s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954183608 (13m15.735116985s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954693472 (13m15.735626841s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954721160 (13m15.735654533s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955219882 (13m15.736153249s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955228357 (13m15.736161733s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955345572 (13m15.736278942s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955694778 (13m15.73662814s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:02.476984010 (13m18.257917395s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:29.758326474 (13m45.53925985s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:48420->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758435603 (13m45.53936898s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:52074->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758514926 (13m45.539448303s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758579547 (13m45.539512941s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 159.223.143.39:38704->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758630446 (13m45.539563824s) [seg_pos={0 0}, upload=208]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:05:29.784351879 (13m45.565285248s) [upload=208]: Done uploading segments... Jul 26 22:05:29.784382292 (13m45.565315669s) [upload=208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.042043774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X..X........XXX.X] duration: 30.026698356s FAIL (2025-07-26 22:06:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026729012s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XX..XXX..X..XX..X.] duration: 494.700361ms FAIL (2025-07-26 22:05:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 26 22:04:39.324461853 (12m55.105395215s) [noise=true , address=104.36.82.14:4223 , piece_id=IMO2TU5HFL3V7DBP4L6J7FNXKNLGG4S2CF5S5FFYRH6N5GP6NRPQ, seg_pos={0 0}, upload=202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 159.223.143.39:47066->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:39.324475303 (12m55.105408689s) [seg_pos={0 0}, upload=202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:04:39.324563844 (12m55.105497224s) [upload=202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:04:39.324568956 (12m55.105502339s) [upload=202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:04:39.462300657 (12m55.243234039s) [upload=202]: Done uploading segments... Jul 26 22:04:39.462336869 (12m55.243270249s) [upload=202]: Upload finished. err: <nil> Jul 26 22:04:59.791460713 (13m15.572394091s) [upload=208]: Starting upload Jul 26 22:04:59.791550369 (13m15.572483756s) [upload=208]: Done starting upload Jul 26 22:04:59.791719355 (13m15.572652723s) [upload=208]: Uploading segments... Jul 26 22:04:59.791892875 (13m15.572826261s) [upload=208]: Got next segment. Inline: false Jul 26 22:04:59.791950644 (13m15.572884027s) [upload=208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:04:59.791959841 (13m15.572893221s) [upload=208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:04:59.820721780 (13m15.601655158s) [seg_pos={0 0}, upload=208]: Begin upload segment... Jul 26 22:04:59.820785657 (13m15.601719027s) [seg_pos={0 0}, upload=208]: Done begin upload segment. Jul 26 22:04:59.820827367 (13m15.601760746s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821370184 (13m15.602303679s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821775197 (13m15.602708574s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822140231 (13m15.603073612s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822522075 (13m15.603455452s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822895726 (13m15.603829106s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828346851 (13m15.609280241s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828835238 (13m15.609768619s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829217860 (13m15.610151245s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829597682 (13m15.610531063s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830018233 (13m15.610951615s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830405152 (13m15.611338532s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830780717 (13m15.611714095s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831187997 (13m15.612121374s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831568937 (13m15.612502319s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831971236 (13m15.612904619s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.941095443 (13m15.722028823s) [upload=208]: Next returned nil segment Jul 26 22:04:59.941117988 (13m15.722051374s) [upload=208]: Waiting for error group managing segments... Jul 26 22:04:59.952061597 (13m15.732994966s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.952764229 (13m15.733697607s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953403179 (13m15.734336555s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953649158 (13m15.734582524s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954183608 (13m15.735116985s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954693472 (13m15.735626841s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954721160 (13m15.735654533s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955219882 (13m15.736153249s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955228357 (13m15.736161733s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955345572 (13m15.736278942s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955694778 (13m15.73662814s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:02.476984010 (13m18.257917395s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:29.758326474 (13m45.53925985s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:48420->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758435603 (13m45.53936898s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:52074->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758514926 (13m45.539448303s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758579547 (13m45.539512941s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 159.223.143.39:38704->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758630446 (13m45.539563824s) [seg_pos={0 0}, upload=208]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:05:29.784351879 (13m45.565285248s) [upload=208]: Done uploading segments... Jul 26 22:05:29.784382292 (13m45.565315669s) [upload=208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.727977ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.025487637s FAIL (2025-07-26 22:05:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== BLVBW5ILRUB5OT57APSJQ7EP6EGZ5A, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.210414100 (12m51.991347489s) [noise=true , address=79.127.219.38:4005 , piece_id=BQXW3FTC4RJLHVZ6QCHYJEE3X3VQXTXBZUGQ5ZGTOVTQV5S4ZOSA, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.210870325 (12m51.991803713s) [noise=true , address=104.36.82.47:4518 , piece_id=HQM7KCLFEYOW3YLRU2RYTCFMITS3ITY3HLZV3HW7WWHYMPJBQYXQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.210884348 (12m51.991817732s) [noise=true , address=104.36.82.46:4503 , piece_id=B4JZ536EMCXWZ6ZOM5NMR5AXFUBUREQMOM3IOLECFAAM6BKHQZVQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.211322418 (12m51.992255798s) [noise=true , address=104.36.82.54:4170 , piece_id=G3DQG3VNJEXZ2Q2UA73P4AD7UPKAZS66JWZBBR346FIL23JADSNQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.211614030 (12m51.992547423s) [noise=true , address=104.36.82.4:4821 , piece_id=NRAIJ3JXBBS76GUH4BY6KZTTX222JF2LKTB3ZWAV7FFQPGIXVEMQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.211735948 (12m51.992669338s) [noise=true , address=104.36.82.38:4697 , piece_id=KOJO25F2SLDWXKBMZVUFVU744JT2YXRPDRHNYRHN73CIB2F2LSXQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.212098020 (12m51.993031408s) [noise=true , address=104.36.82.10:4106 , piece_id=ZBQXSEDO74OHTE2N4EJJ7K3TNJ62UBDZJOQCKLL5CUZHEWCNXVZQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.212431113 (12m51.993364501s) [noise=true , address=104.36.82.14:4224 , piece_id=QPVTUI52LQ7WDPBS5UCHBN3XQ7HGREG45ZPNSQ6OBY6WQELLRIOQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.212491628 (12m51.993425014s) [noise=true , address=104.36.82.56:4256 , piece_id=DWEH2KOYOM3RR56PAQLUULTRJTLDYSPOI4XAH3P5V7VV7XDJUVUQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.212916342 (12m51.993849726s) [noise=true , address=79.127.201.215:4345 , piece_id=6ZI2KRSUUKIYOWXAV7SSV5IUWV5ONNDTNV2TV7USGQWIZHNNXWFA, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.213298650 (12m51.994232038s) [noise=true , address=79.127.205.234:4608 , piece_id=O2DQWA4DTIP45Y2477QL5626UMEYJJXBCFNQ54IY6CILSAMC7KMQ, seg_pos={0 0}, upload=204]: Uploading piece... Jul 26 22:04:36.315822167 (12m52.096755537s) [upload=204]: Next returned nil segment Jul 26 22:04:36.316433454 (12m52.097366842s) [upload=204]: Waiting for error group managing segments... Jul 26 22:04:36.338009830 (12m52.118943196s) [noise=true , address=79.127.219.38:4005 , piece_id=BQXW3FTC4RJLHVZ6QCHYJEE3X3VQXTXBZUGQ5ZGTOVTQV5S4ZOSA, seg_pos={0 0}, upload=204]: Done uploading piece. err: <nil> Jul 26 22:04:36.554744133 (12m52.335677519s) [noise=true , address=79.127.220.98:4402 , piece_id=GHIAKFV26HUPNRRB2BYK4G2OZMMUGIRE766OSA7FKDJMGQE3CVUQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: <nil> Jul 26 22:04:36.600389952 (12m52.381323336s) [noise=true , address=64.49.13.138:4482 , piece_id=2O7IB5ADHJRIAVLQVASWOVQBWV2EAQPI7RFLDKXIKR3ODTS6V7YA, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:04:36.884101216 (12m52.665034601s) [noise=true , address=79.127.205.245:4242 , piece_id=KF4BBYQ4COVVCPY2DO2Y7ABLVBW5ILRUB5OT57APSJQ7EP6EGZ5A, seg_pos={0 0}, upload=204]: Done uploading piece. err: <nil> Jul 26 22:04:37.380917066 (12m53.161850429s) [noise=true , address=79.127.205.234:4608 , piece_id=O2DQWA4DTIP45Y2477QL5626UMEYJJXBCFNQ54IY6CILSAMC7KMQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: <nil> Jul 26 22:04:39.755085337 (12m55.536018698s) [noise=true , address=79.127.201.215:4345 , piece_id=6ZI2KRSUUKIYOWXAV7SSV5IUWV5ONNDTNV2TV7USGQWIZHNNXWFA, seg_pos={0 0}, upload=204]: Done uploading piece. err: <nil> Jul 26 22:05:36.141865164 (13m51.92279855s) [noise=true , address=104.36.82.14:4224 , piece_id=QPVTUI52LQ7WDPBS5UCHBN3XQ7HGREG45ZPNSQ6OBY6WQELLRIOQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: write tcp 159.223.143.39:35946->104.36.82.14:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.141955143 (13m51.92288852s) [noise=true , address=104.36.82.13:4206 , piece_id=JPAKOPSI5YPA3ZZNLSD4SBOA6PAHM5ATHHBBPG2GD2NR3EZAA7PA, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:1ih3m2bU4PfDMX5zfXoGw3t2eyY1u6wVnpqczo2sz7aTq99R9T, address:104.36.82.13:4206): protocol: write tcp 159.223.143.39:34320->104.36.82.13:4206: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142040848 (13m51.92297423s) [noise=true , address=104.36.82.46:4503 , piece_id=B4JZ536EMCXWZ6ZOM5NMR5AXFUBUREQMOM3IOLECFAAM6BKHQZVQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 159.223.143.39:42734->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142087592 (13m51.923020955s) [noise=true , address=104.36.82.38:4697 , piece_id=KOJO25F2SLDWXKBMZVUFVU744JT2YXRPDRHNYRHN73CIB2F2LSXQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:1mLiUHc5wogTx5spRu3aNfAh35g31kwXbEfn5aCJoHXUVDkhGU, address:104.36.82.38:4697): protocol: write tcp 159.223.143.39:58724->104.36.82.38:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142176624 (13m51.923110005s) [noise=true , address=104.36.82.47:4518 , piece_id=HQM7KCLFEYOW3YLRU2RYTCFMITS3ITY3HLZV3HW7WWHYMPJBQYXQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: write tcp 159.223.143.39:41942->104.36.82.47:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142260033 (13m51.923193422s) [noise=true , address=104.36.82.54:4170 , piece_id=G3DQG3VNJEXZ2Q2UA73P4AD7UPKAZS66JWZBBR346FIL23JADSNQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 159.223.143.39:42724->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142333772 (13m51.92326715s) [noise=true , address=104.36.82.52:4101 , piece_id=4ID65TNJQ7PT22AA3REJDHUSBC7KWPH7Q45PHGJPYFUGXB6ASGYA, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:53078->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142441450 (13m51.923374829s) [noise=true , address=104.36.82.17:4334 , piece_id=QL5UNX5VYM4GZN6PQTZCSETZDEZVPVTMDG4YQPG2BBQ4Y5PH2CHQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:104.36.82.17:4334): protocol: write tcp 159.223.143.39:50714->104.36.82.17:4334: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142510202 (13m51.92344358s) [noise=true , address=104.36.82.4:4821 , piece_id=NRAIJ3JXBBS76GUH4BY6KZTTX222JF2LKTB3ZWAV7FFQPGIXVEMQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: write tcp 159.223.143.39:56846->104.36.82.4:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142586043 (13m51.923519408s) [noise=true , address=104.36.82.55:4228 , piece_id=QU7DZ65BMPI4LRNBQZASTWSPPNDAAA7OB5AETCJE4T4AVKZB3RCA, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 159.223.143.39:35012->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142626400 (13m51.923559777s) [noise=true , address=104.36.82.10:4106 , piece_id=ZBQXSEDO74OHTE2N4EJJ7K3TNJ62UBDZJOQCKLL5CUZHEWCNXVZQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:12CsQq2BuV9hNc5fVAsw882NkDJMEXZeNwhV2FGNGKAU6wuFBHm, address:104.36.82.10:4106): protocol: write tcp 159.223.143.39:55610->104.36.82.10:4106: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:36.142740162 (13m51.923673542s) [noise=true , address=104.36.82.56:4256 , piece_id=DWEH2KOYOM3RR56PAQLUULTRJTLDYSPOI4XAH3P5V7VV7XDJUVUQ, seg_pos={0 0}, upload=204]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: write tcp 159.223.143.39:36414->104.36.82.56:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 22:05:36.142764577 (13m51.923697962s) [seg_pos={0 0}, upload=204]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:05:36.165417757 (13m51.946351126s) [upload=204]: Done uploading segments... Jul 26 22:05:36.165444902 (13m51.946378278s) [upload=204]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.025513543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XX..XXX..X..XX..X] duration: 30.028142067s FAIL (2025-07-26 22:05:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 26 22:04:39.324461853 (12m55.105395215s) [noise=true , address=104.36.82.14:4223 , piece_id=IMO2TU5HFL3V7DBP4L6J7FNXKNLGG4S2CF5S5FFYRH6N5GP6NRPQ, seg_pos={0 0}, upload=202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 159.223.143.39:47066->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:39.324475303 (12m55.105408689s) [seg_pos={0 0}, upload=202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:04:39.324563844 (12m55.105497224s) [upload=202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:04:39.324568956 (12m55.105502339s) [upload=202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:04:39.462300657 (12m55.243234039s) [upload=202]: Done uploading segments... Jul 26 22:04:39.462336869 (12m55.243270249s) [upload=202]: Upload finished. err: <nil> Jul 26 22:04:59.791460713 (13m15.572394091s) [upload=208]: Starting upload Jul 26 22:04:59.791550369 (13m15.572483756s) [upload=208]: Done starting upload Jul 26 22:04:59.791719355 (13m15.572652723s) [upload=208]: Uploading segments... Jul 26 22:04:59.791892875 (13m15.572826261s) [upload=208]: Got next segment. Inline: false Jul 26 22:04:59.791950644 (13m15.572884027s) [upload=208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:04:59.791959841 (13m15.572893221s) [upload=208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:04:59.820721780 (13m15.601655158s) [seg_pos={0 0}, upload=208]: Begin upload segment... Jul 26 22:04:59.820785657 (13m15.601719027s) [seg_pos={0 0}, upload=208]: Done begin upload segment. Jul 26 22:04:59.820827367 (13m15.601760746s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821370184 (13m15.602303679s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.821775197 (13m15.602708574s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822140231 (13m15.603073612s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822522075 (13m15.603455452s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.822895726 (13m15.603829106s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828346851 (13m15.609280241s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.828835238 (13m15.609768619s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829217860 (13m15.610151245s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.829597682 (13m15.610531063s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830018233 (13m15.610951615s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830405152 (13m15.611338532s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.830780717 (13m15.611714095s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831187997 (13m15.612121374s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831568937 (13m15.612502319s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.831971236 (13m15.612904619s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Uploading piece... Jul 26 22:04:59.941095443 (13m15.722028823s) [upload=208]: Next returned nil segment Jul 26 22:04:59.941117988 (13m15.722051374s) [upload=208]: Waiting for error group managing segments... Jul 26 22:04:59.952061597 (13m15.732994966s) [noise=true , address=45.91.116.11:4117 , piece_id=2Z7WNKQIJD2KJ3TXH7BDODPS2QOHXUNMK6XIRS22OY5XKRQ5PQUQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.952764229 (13m15.733697607s) [noise=true , address=45.91.116.14:4190 , piece_id=BTERJPGADAAONHEZ2KAHCGZCU6NQV5FAGNXH3JQDVMRR7FCIPL4A, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953403179 (13m15.734336555s) [noise=true , address=45.91.116.9:4261 , piece_id=3I7FSUCC3Y2E4J6BWCNK5V63GWSBG2S3KV4PGH4U5MN3BHBO2JVA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.953649158 (13m15.734582524s) [noise=true , address=45.91.116.9:4262 , piece_id=3QEEGMX4AI6IKZEFFXBKCQQQ6AA7ENZMGS73UAGTSIMB7PU7XOMQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954183608 (13m15.735116985s) [noise=true , address=45.91.116.13:4154 , piece_id=XA3IDQBFMD2JPUC6THO7FGK2KAZYUBGXGEBFHFA5UVQNRAGXRHSQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954693472 (13m15.735626841s) [noise=true , address=45.91.116.8:4047 , piece_id=SKCVFSYRC4IOOVJNMTR446TYF6VEAXF6MWPMJCTULRWTRXBGDDGA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.954721160 (13m15.735654533s) [noise=true , address=45.91.116.4:4938 , piece_id=UCS54QBVZ5ERE6MWNRDVM4JN42RRXTSWAMM5LFCSC4ZDIHQMG4IQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955219882 (13m15.736153249s) [noise=true , address=45.91.116.4:4940 , piece_id=NCLBK5FN3LB6T4XTOTYEKBQJOSM7LSRSEWUJU3ALZJD7W3DORVHA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955228357 (13m15.736161733s) [noise=true , address=45.91.116.5:4297 , piece_id=6SRGDIZZ2JGAECSXSR6MYIQDDOQJJI6YCEDELOU5KXUYKXFB5GMA, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955345572 (13m15.736278942s) [noise=true , address=45.91.116.6:4973 , piece_id=LRYATFRDJFMIAATQTQ4AL24CB3XIWZIPZLMHXD3RBCYF2UB5JR4Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:04:59.955694778 (13m15.73662814s) [noise=true , address=45.91.116.11:4119 , piece_id=QGHDPCNAMVRC2PRKOOZSXSGA2RTXLHSEZQJEF6WY6BD43MYNX2KQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:02.476984010 (13m18.257917395s) [noise=true , address=77.90.31.14:4662 , piece_id=A74ZIJH7AP24PTR2WGUOLWETJCW4UHPCJMJO4IF6VGOD4T2SO4VQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: <nil> Jul 26 22:05:29.758326474 (13m45.53925985s) [noise=true , address=104.36.82.10:4077 , piece_id=LGTGUQVCX6YOR5GG6XEAFFJKUS73JFOEVPLFDDJHPUTFSPYHFN2Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:48420->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758435603 (13m45.53936898s) [noise=true , address=104.36.82.15:4258 , piece_id=4Q3AZ5LNQI43PHXOTULN3Z5NPBDRQ35PE3YNX4WDVGGBWV35343Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:52074->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758514926 (13m45.539448303s) [noise=true , address=104.36.82.7:4911 , piece_id=OJPS4GCQIOQGYGI2USJVZNQZZD7OPOQRTQGWYYUPSCW2APKFSHEQ, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758579547 (13m45.539512941s) [noise=true , address=104.36.82.8:4970 , piece_id=FC2UA6SGJC73MU66CE3Y3UH754PS5PXSKRDIBANEO26QR4XIIK7Q, seg_pos={0 0}, upload=208]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 159.223.143.39:38704->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:05:29.758630446 (13m45.539563824s) [seg_pos={0 0}, upload=208]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:05:29.784351879 (13m45.565285248s) [upload=208]: Done uploading segments... Jul 26 22:05:29.784382292 (13m45.565315669s) [upload=208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028165426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................X..X........XXX.] duration: 14.477193527s FAIL (2025-07-26 22:05:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.477227395s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X..X........XXX] duration: 30.036230491s FAIL (2025-07-26 22:05:03 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.036264479s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XX..XXX..X..XX..] duration: 10.850757785s FAIL (2025-07-26 22:04:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .36.82.6:4874 , piece_id=SN4VQBEGUOPLFC5GCZAVBGTWVBCWJ5NHAI4AZH6C3JMPKRPLZEBA, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:56750->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.225017349 (11m9.005950734s) [seg_pos={0 0}, upload=167]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:02:53.247189426 (11m9.028122802s) [upload=167]: Done uploading segments... Jul 26 22:02:53.247221529 (11m9.028154905s) [upload=167]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:02:56.462287386 (11m12.243220749s) [upload=177]: Starting upload Jul 26 22:02:56.462366615 (11m12.243299991s) [upload=177]: Done starting upload Jul 26 22:02:56.462701194 (11m12.24363457s) [upload=177]: Uploading segments... Jul 26 22:02:56.462767132 (11m12.243700509s) [upload=177]: Got next segment. Inline: false Jul 26 22:02:56.462809229 (11m12.243742607s) [upload=177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:02:56.462815121 (11m12.243748499s) [upload=177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:02:56.495558533 (11m12.276492161s) [seg_pos={0 0}, upload=177]: Begin upload segment... Jul 26 22:02:56.495614233 (11m12.276547669s) [seg_pos={0 0}, upload=177]: Done begin upload segment. Jul 26 22:02:56.495647196 (11m12.276580582s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.495884190 (11m12.276817582s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496275228 (11m12.277208617s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496408089 (11m12.277341472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496668774 (11m12.277602166s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496871365 (11m12.277804743s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497026953 (11m12.277960343s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497346995 (11m12.278280376s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497445130 (11m12.27837852s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497840356 (11m12.278773746s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497848583 (11m12.278781967s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498256649 (11m12.279190031s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498328617 (11m12.279262005s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498649087 (11m12.279582472s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498773449 (11m12.279706817s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.499220962 (11m12.280154343s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.605981739 (11m12.386915116s) [upload=177]: Next returned nil segment Jul 26 22:02:56.606001640 (11m12.386935025s) [upload=177]: Waiting for error group managing segments... Jul 26 22:02:56.610808323 (11m12.391741715s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.611449666 (11m12.392383043s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612914089 (11m12.393847472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612977922 (11m12.393911289s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613285867 (11m12.394219244s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613567443 (11m12.394500816s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613877717 (11m12.394811096s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614080230 (11m12.3950136s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614645429 (11m12.39557882s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614687861 (11m12.395621241s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:05.434706200 (11m21.215639563s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:26.432313509 (11m42.213246898s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:34166->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432428401 (11m42.213361789s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:41320->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432451409 (11m42.213384789s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:36250->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432528838 (11m42.213462205s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:50228->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432553381 (11m42.213486767s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:38664->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432601674 (11m42.213535062s) [seg_pos={0 0}, upload=177]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:03:26.464914041 (11m42.245847422s) [upload=177]: Done uploading segments... Jul 26 22:03:26.464946132 (11m42.245879512s) [upload=177]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.850787424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.047438494s FAIL (2025-07-26 22:04:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== ZJPGI5RQP5Q2JB3NIKVOG5FYKKECENQBA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.095588469 (11m51.876521856s) [noise=true , address=77.90.31.11:4574 , piece_id=JMLDGKMMU5BNP5QK6WHCT2ARIGSIH7WT4H4I7CB7C7JP6X3RX5HA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.098828041 (11m51.87976143s) [noise=true , address=77.90.31.19:4871 , piece_id=CWU62BSAYKMZMDQTTDZBK5TC4WBMVSXGGGOTGUXMH5QROMNE75RA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.102023111 (11m51.882956477s) [noise=true , address=79.127.205.245:4237 , piece_id=QGVTRMKCC7IE2TZWE24JBXW2HUA4AKNL5ULYEXLHCQWX2L4ECOYA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.104479899 (11m51.885413289s) [noise=true , address=104.36.82.51:4055 , piece_id=B373ELBCT4KSNN3LZ33Z7PNKMS2WDHPKYOFUSURXYTJR7IXP7HGA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.107463175 (11m51.888396783s) [noise=true , address=104.36.82.46:4499 , piece_id=FWXMYGVNQTFDKZJZ6RGGQZH3JNAEV745YQNH3MQ3OZMW5JLTGWOQ, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.109276866 (11m51.890210254s) [noise=true , address=104.36.82.53:4145 , piece_id=K6XI2BBTWZDY5ZI5VSOGDUXBKWPBZ7YHYXDGNQFBHZTW3T4UYT3Q, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.122936435 (11m51.903869825s) [noise=true , address=104.36.82.21:4476 , piece_id=LO5HUHCSY447NBQBC6CA65HUKP4TPIRGTN45OPEJEC2HEN7KXDZA, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.124276030 (11m51.905209419s) [noise=true , address=79.127.201.215:4351 , piece_id=LDURB42FUIAF2VLYCITKSDOGGUOSLGZWGYUGEHPUKD3LR54LM3KQ, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.126250085 (11m51.907183472s) [noise=true , address=104.36.82.8:4977 , piece_id=PDWLOJUPRRP2LWK3KU3UHRTDPFHQFAZQSVIWVBKCIKMEUY66ZIZQ, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.129643608 (11m51.910576982s) [noise=true , address=104.36.82.38:4698 , piece_id=XHVLTFYUDFBCC7KAJWAFL5UNERCIAW63CSNREEP7UE53TJ2EBYBQ, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.132579720 (11m51.913513107s) [noise=true , address=104.36.82.7:4919 , piece_id=WJZ7FDTBJ4YS5K3LEDYK4NPLSOAT2TKATJGTRRITUV6GHPQBHX3A, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.135523591 (11m51.91645711s) [noise=true , address=45.91.116.14:4222 , piece_id=UI2W3JHW62ANLNLEQBAQPC7257AFT255XNLYVP6KSO4WHJ6K35FQ, seg_pos={0 0}, upload=189]: Uploading piece... Jul 26 22:03:36.230291826 (11m52.011225195s) [upload=189]: Next returned nil segment Jul 26 22:03:36.230319135 (11m52.011252529s) [upload=189]: Waiting for error group managing segments... Jul 26 22:03:36.711434118 (11m52.492367496s) [noise=true , address=79.127.205.245:4237 , piece_id=QGVTRMKCC7IE2TZWE24JBXW2HUA4AKNL5ULYEXLHCQWX2L4ECOYA, seg_pos={0 0}, upload=189]: Done uploading piece. err: <nil> Jul 26 22:03:38.083415068 (11m53.864348455s) [noise=true , address=45.91.116.14:4222 , piece_id=UI2W3JHW62ANLNLEQBAQPC7257AFT255XNLYVP6KSO4WHJ6K35FQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:03:39.016904002 (11m54.797837393s) [noise=true , address=77.90.31.11:4574 , piece_id=JMLDGKMMU5BNP5QK6WHCT2ARIGSIH7WT4H4I7CB7C7JP6X3RX5HA, seg_pos={0 0}, upload=189]: Done uploading piece. err: <nil> Jul 26 22:03:39.848539590 (11m55.629472958s) [noise=true , address=77.90.31.19:4871 , piece_id=CWU62BSAYKMZMDQTTDZBK5TC4WBMVSXGGGOTGUXMH5QROMNE75RA, seg_pos={0 0}, upload=189]: Done uploading piece. err: <nil> Jul 26 22:03:41.908248927 (11m57.689182302s) [noise=true , address=79.127.219.47:4061 , piece_id=CYAD5FLGXI6B3VDBCFMECCCPKVCZVDSPVCF5H5B33TR6KS2BOIDQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: <nil> Jul 26 22:03:46.318778746 (12m2.099712115s) [noise=true , address=79.127.201.215:4351 , piece_id=LDURB42FUIAF2VLYCITKSDOGGUOSLGZWGYUGEHPUKD3LR54LM3KQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: <nil> Jul 26 22:04:36.007451274 (12m51.788384674s) [noise=true , address=104.36.82.36:4632 , piece_id=73OIT2TKNEX5P3VNJCFT67QQDFRPLRVUCSR7WQ4HNLK6J6GAEAGQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: write tcp 159.223.143.39:51616->104.36.82.36:4632: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007549152 (12m51.788482526s) [noise=true , address=104.36.82.52:4091 , piece_id=MURVXID55L64GMEFXYHZJPGI5RQP5Q2JB3NIKVOG5FYKKECENQBA, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: write tcp 159.223.143.39:34254->104.36.82.52:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007630675 (12m51.788564041s) [noise=true , address=104.36.82.8:4977 , piece_id=PDWLOJUPRRP2LWK3KU3UHRTDPFHQFAZQSVIWVBKCIKMEUY66ZIZQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: write tcp 159.223.143.39:60990->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007715706 (12m51.788649081s) [noise=true , address=104.36.82.16:4321 , piece_id=C2E4XB22KKVNSL3T6BZ3BOL3HIUBUT72WDNZZUDHNAFWBPIS7T4Q, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae, address:104.36.82.16:4321): protocol: write tcp 159.223.143.39:57314->104.36.82.16:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007798034 (12m51.788731398s) [noise=true , address=104.36.82.46:4499 , piece_id=FWXMYGVNQTFDKZJZ6RGGQZH3JNAEV745YQNH3MQ3OZMW5JLTGWOQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 159.223.143.39:45212->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007866010 (12m51.788799373s) [noise=true , address=104.36.82.53:4145 , piece_id=K6XI2BBTWZDY5ZI5VSOGDUXBKWPBZ7YHYXDGNQFBHZTW3T4UYT3Q, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 159.223.143.39:34246->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007928349 (12m51.788861715s) [noise=true , address=104.36.82.38:4698 , piece_id=XHVLTFYUDFBCC7KAJWAFL5UNERCIAW63CSNREEP7UE53TJ2EBYBQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:12hbvyP68MKUCvYEhi1kFD33zSRf4oxJgH6weHRnYiYKX9QEC1Q, address:104.36.82.38:4698): protocol: write tcp 159.223.143.39:46352->104.36.82.38:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.007998177 (12m51.788931553s) [noise=true , address=104.36.82.21:4476 , piece_id=LO5HUHCSY447NBQBC6CA65HUKP4TPIRGTN45OPEJEC2HEN7KXDZA, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:12NKwRvjiEpY7DrbqFGaDHQ8no5YV4cUALnpsvQ8icUPoLvMYen, address:104.36.82.21:4476): protocol: write tcp 159.223.143.39:49028->104.36.82.21:4476: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.008084601 (12m51.789017981s) [noise=true , address=104.36.82.2:4761 , piece_id=RBRZMJXUFZ5ENPQDNXUXUD6MQT4POJ4EKZLIDZHE6MFL6HIBLBDQ, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:104.36.82.2:4761): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:04:36.008243512 (12m51.789176894s) [noise=true , address=104.36.82.55:4220 , piece_id=JX4HXNGZZZ7BSHBMV3K7R3ZEMJCO6BK465VTFK56WVCEBT4ZONMA, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:04:36.008292493 (12m51.789225859s) [noise=true , address=104.36.82.7:4919 , piece_id=WJZ7FDTBJ4YS5K3LEDYK4NPLSOAT2TKATJGTRRITUV6GHPQBHX3A, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w, address:104.36.82.7:4919): protocol: write tcp 159.223.143.39:40964->104.36.82.7:4919: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.008303952 (12m51.789237315s) [noise=true , address=104.36.82.51:4055 , piece_id=B373ELBCT4KSNN3LZ33Z7PNKMS2WDHPKYOFUSURXYTJR7IXP7HGA, seg_pos={0 0}, upload=189]: Done uploading piece. err: ecclient: upload failed (node:1SAWKJT2JEb9ReJUGDZ545Rp7LeQoCJKYsQgmrDA9csufPf8us, address:104.36.82.51:4055): protocol: write tcp 159.223.143.39:34300->104.36.82.51:4055: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:04:36.008332288 (12m51.789265675s) [seg_pos={0 0}, upload=189]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:04:36.051016589 (12m51.831949953s) [upload=189]: Done uploading segments... Jul 26 22:04:36.051071041 (12m51.832004419s) [upload=189]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.047487259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X..X........XX] duration: 30.029774939s FAIL (2025-07-26 22:03:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02979923s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................................XX..XXX..X..XX.] duration: 5.581465415s FAIL (2025-07-26 22:03:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .36.82.6:4874 , piece_id=SN4VQBEGUOPLFC5GCZAVBGTWVBCWJ5NHAI4AZH6C3JMPKRPLZEBA, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:56750->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.225017349 (11m9.005950734s) [seg_pos={0 0}, upload=167]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:02:53.247189426 (11m9.028122802s) [upload=167]: Done uploading segments... Jul 26 22:02:53.247221529 (11m9.028154905s) [upload=167]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:02:56.462287386 (11m12.243220749s) [upload=177]: Starting upload Jul 26 22:02:56.462366615 (11m12.243299991s) [upload=177]: Done starting upload Jul 26 22:02:56.462701194 (11m12.24363457s) [upload=177]: Uploading segments... Jul 26 22:02:56.462767132 (11m12.243700509s) [upload=177]: Got next segment. Inline: false Jul 26 22:02:56.462809229 (11m12.243742607s) [upload=177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:02:56.462815121 (11m12.243748499s) [upload=177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:02:56.495558533 (11m12.276492161s) [seg_pos={0 0}, upload=177]: Begin upload segment... Jul 26 22:02:56.495614233 (11m12.276547669s) [seg_pos={0 0}, upload=177]: Done begin upload segment. Jul 26 22:02:56.495647196 (11m12.276580582s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.495884190 (11m12.276817582s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496275228 (11m12.277208617s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496408089 (11m12.277341472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496668774 (11m12.277602166s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496871365 (11m12.277804743s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497026953 (11m12.277960343s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497346995 (11m12.278280376s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497445130 (11m12.27837852s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497840356 (11m12.278773746s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497848583 (11m12.278781967s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498256649 (11m12.279190031s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498328617 (11m12.279262005s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498649087 (11m12.279582472s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498773449 (11m12.279706817s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.499220962 (11m12.280154343s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.605981739 (11m12.386915116s) [upload=177]: Next returned nil segment Jul 26 22:02:56.606001640 (11m12.386935025s) [upload=177]: Waiting for error group managing segments... Jul 26 22:02:56.610808323 (11m12.391741715s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.611449666 (11m12.392383043s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612914089 (11m12.393847472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612977922 (11m12.393911289s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613285867 (11m12.394219244s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613567443 (11m12.394500816s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613877717 (11m12.394811096s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614080230 (11m12.3950136s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614645429 (11m12.39557882s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614687861 (11m12.395621241s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:05.434706200 (11m21.215639563s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:26.432313509 (11m42.213246898s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:34166->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432428401 (11m42.213361789s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:41320->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432451409 (11m42.213384789s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:36250->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432528838 (11m42.213462205s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:50228->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432553381 (11m42.213486767s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:38664->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432601674 (11m42.213535062s) [seg_pos={0 0}, upload=177]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:03:26.464914041 (11m42.245847422s) [upload=177]: Done uploading segments... Jul 26 22:03:26.464946132 (11m42.245879512s) [upload=177]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.5814885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.029066105s FAIL (2025-07-26 22:03:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== D6JBHOYU47WSWWRG56YTKQPWILAQ, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.963121048 (10m47.744054436s) [seg_pos={0 0}, upload=171]: Done begin upload segment. Jul 26 22:02:31.963161087 (10m47.744094474s) [noise=true , address=79.127.219.51:4210 , piece_id=G7YQRSHGI5VN5OOPSZZLUZBSVP2JA32T4KM5GZ3JBM6K2OBT2PJQ, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.963539292 (10m47.744472681s) [noise=true , address=104.36.82.15:4265 , piece_id=5UHM5GNL3QEU2ICWE6DON7ODNKNZXRDYMHGNDFOUKYVJ44NRBKYQ, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.963928921 (10m47.744862312s) [noise=true , address=156.146.43.225:4449 , piece_id=HO6JU5GS3SEH73AYWVKVCK4X5LOIHT3NP6OUDO3LPMN2NWQK3Q3Q, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.964317286 (10m47.745250673s) [noise=true , address=104.36.82.51:4051 , piece_id=KV6R7MAL7P3ADEAG26QQHUGFN6AH7IVEZ2QLLVVYNHPTTUWDHYKA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.965126003 (10m47.746059393s) [noise=true , address=104.36.82.47:4528 , piece_id=HSM4VBB2UIXV4LZYQIMXGJQK2YYP5DDVDYDD32N2G2LH7ZIIWWEA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.965517351 (10m47.74645074s) [noise=true , address=104.36.82.2:4736 , piece_id=Z6J4Q5QSRVFRKVNYALXKABTBIEK6RSAWHS6MTDTVXHFGHHI2Y4AA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.966057133 (10m47.746990524s) [noise=true , address=104.36.82.54:4157 , piece_id=OCQP2Z3XEAKPDA4QDX5ZT5LT44USLIZOMKS5226R7JGOGFGFG5GQ, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.966444369 (10m47.747377756s) [noise=true , address=104.36.82.46:4496 , piece_id=ENTCZEKFTDAXAR5NC3UNAPMNOVRPI7SSFSDQA67QHY7KAYJ3YHYQ, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.966961110 (10m47.747894498s) [noise=true , address=79.127.201.215:4355 , piece_id=HPSD3KDSDJVDLLPO6ZUPN3X7SONPTVDZDJUEZSDF2BXXE2TLZWVA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.967396396 (10m47.748329784s) [noise=true , address=156.146.43.225:4448 , piece_id=CIZ6TSOEV7NKLL6ZBVFG6IAN74W2VZHZ67RDVGQS23C2PAY2IXBA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.967777825 (10m47.748711216s) [noise=true , address=45.91.116.8:4065 , piece_id=LLGAMGMFBENTOB2IEZ62H7SRVENPRYH3W6H5ZVGSA27B5JMTRPXA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:31.968202377 (10m47.749135772s) [noise=true , address=104.36.82.5:4854 , piece_id=TAASH2NRYJAMB2FQJ2YPCN7GGWTKQNI5KYIBM35J23QADKCVR6SA, seg_pos={0 0}, upload=171]: Uploading piece... Jul 26 22:02:32.064347022 (10m47.845280404s) [upload=171]: Next returned nil segment Jul 26 22:02:32.064371149 (10m47.845304547s) [upload=171]: Waiting for error group managing segments... Jul 26 22:02:32.880014718 (10m48.660948096s) [noise=true , address=45.91.116.8:4065 , piece_id=LLGAMGMFBENTOB2IEZ62H7SRVENPRYH3W6H5ZVGSA27B5JMTRPXA, seg_pos={0 0}, upload=171]: Done uploading piece. err: <nil> Jul 26 22:02:33.725959719 (10m49.506893084s) [noise=true , address=79.127.219.51:4210 , piece_id=G7YQRSHGI5VN5OOPSZZLUZBSVP2JA32T4KM5GZ3JBM6K2OBT2PJQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: <nil> Jul 26 22:02:37.839631294 (10m53.620564678s) [noise=true , address=156.146.43.225:4448 , piece_id=CIZ6TSOEV7NKLL6ZBVFG6IAN74W2VZHZ67RDVGQS23C2PAY2IXBA, seg_pos={0 0}, upload=171]: Done uploading piece. err: <nil> Jul 26 22:02:38.376857867 (10m54.157791254s) [noise=true , address=156.146.43.225:4449 , piece_id=HO6JU5GS3SEH73AYWVKVCK4X5LOIHT3NP6OUDO3LPMN2NWQK3Q3Q, seg_pos={0 0}, upload=171]: Done uploading piece. err: <nil> Jul 26 22:02:43.551375490 (10m59.332308856s) [noise=true , address=79.127.201.215:4355 , piece_id=HPSD3KDSDJVDLLPO6ZUPN3X7SONPTVDZDJUEZSDF2BXXE2TLZWVA, seg_pos={0 0}, upload=171]: Done uploading piece. err: <nil> Jul 26 22:03:31.881535005 (11m47.662468475s) [noise=true , address=104.36.82.2:4736 , piece_id=Z6J4Q5QSRVFRKVNYALXKABTBIEK6RSAWHS6MTDTVXHFGHHI2Y4AA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:104.36.82.2:4736): protocol: write tcp 159.223.143.39:36232->104.36.82.2:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882110921 (11m47.663044342s) [noise=true , address=104.36.82.52:4118 , piece_id=CZBXA5N44VCEWJO5A22T5EJGZOHG7JRBMW4FOQ6B25LIPLFTURSQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: rpc: dial tcp 104.36.82.52:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882197265 (11m47.663130642s) [noise=true , address=104.36.82.22:4529 , piece_id=5D3F445BQ7S5M6XGIFWQ2HHCD6JBHOYU47WSWWRG56YTKQPWILAQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:12bNgTA4CY9ShB6Mi3QpceTEqNCr8ohCMHNVvU6S1qJp2TdVP1i, address:104.36.82.22:4529): protocol: rpc: dial tcp 104.36.82.22:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882269075 (11m47.66320244s) [noise=true , address=104.36.82.55:4200 , piece_id=TC4JNOQBGPQUUFG34WYWQ7GHFIUMKLDLQW634TVS6H3JMIU7OLPA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882426814 (11m47.663360238s) [noise=true , address=104.36.82.56:4235 , piece_id=BRUZ4BI4MSFWLTSKPKO7UHLEWXUUTKBS5OP5EKNINVXJYJLG46HA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882564258 (11m47.663497621s) [noise=true , address=104.36.82.15:4265 , piece_id=5UHM5GNL3QEU2ICWE6DON7ODNKNZXRDYMHGNDFOUKYVJ44NRBKYQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:1NscbwYiKarBdony9TwdccuEKg7f4VXEXerFfWmCiat4RKdXFi, address:104.36.82.15:4265): protocol: write tcp 159.223.143.39:58206->104.36.82.15:4265: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.882646322 (11m47.663579701s) [noise=true , address=104.36.82.46:4496 , piece_id=ENTCZEKFTDAXAR5NC3UNAPMNOVRPI7SSFSDQA67QHY7KAYJ3YHYQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: rpc: dial tcp 104.36.82.46:4496: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883062625 (11m47.663996005s) [noise=true , address=104.36.82.5:4854 , piece_id=TAASH2NRYJAMB2FQJ2YPCN7GGWTKQNI5KYIBM35J23QADKCVR6SA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883135746 (11m47.664069107s) [noise=true , address=104.36.82.51:4051 , piece_id=KV6R7MAL7P3ADEAG26QQHUGFN6AH7IVEZ2QLLVVYNHPTTUWDHYKA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:104.36.82.51:4051): protocol: rpc: dial tcp 104.36.82.51:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883162821 (11m47.664096199s) [noise=true , address=104.36.82.53:4152 , piece_id=7LI2NQQCXV3X4LEZIND3L52SOBSBKDWQT7BZYXEIJ4R7OMWJI2NQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: rpc: dial tcp 104.36.82.53:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883252688 (11m47.664186065s) [noise=true , address=104.36.82.20:4451 , piece_id=F6QBP3KXQY2OMGGC7IR6DJBQUHHA75C6IEL4YY7MKVVM6Z5CXFPQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:12fWecKRc63WWDEqfHomHdVKVaH3naVgZx6nhLZDJ4JnL3CfPQ, address:104.36.82.20:4451): protocol: rpc: dial tcp 104.36.82.20:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883267212 (11m47.664200574s) [noise=true , address=104.36.82.47:4528 , piece_id=HSM4VBB2UIXV4LZYQIMXGJQK2YYP5DDVDYDD32N2G2LH7ZIIWWEA, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883313012 (11m47.664246377s) [noise=true , address=104.36.82.54:4157 , piece_id=OCQP2Z3XEAKPDA4QDX5ZT5LT44USLIZOMKS5226R7JGOGFGFG5GQ, seg_pos={0 0}, upload=171]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: write tcp 159.223.143.39:37458->104.36.82.54:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:31.883344859 (11m47.66427824s) [seg_pos={0 0}, upload=171]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:03:31.908051123 (11m47.688984504s) [upload=171]: Done uploading segments... Jul 26 22:03:31.908327039 (11m47.689260428s) [upload=171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029101882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX..XXX..X..XX] duration: 30.035690129s FAIL (2025-07-26 22:03:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .36.82.6:4874 , piece_id=SN4VQBEGUOPLFC5GCZAVBGTWVBCWJ5NHAI4AZH6C3JMPKRPLZEBA, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:56750->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.225017349 (11m9.005950734s) [seg_pos={0 0}, upload=167]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:02:53.247189426 (11m9.028122802s) [upload=167]: Done uploading segments... Jul 26 22:02:53.247221529 (11m9.028154905s) [upload=167]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:02:56.462287386 (11m12.243220749s) [upload=177]: Starting upload Jul 26 22:02:56.462366615 (11m12.243299991s) [upload=177]: Done starting upload Jul 26 22:02:56.462701194 (11m12.24363457s) [upload=177]: Uploading segments... Jul 26 22:02:56.462767132 (11m12.243700509s) [upload=177]: Got next segment. Inline: false Jul 26 22:02:56.462809229 (11m12.243742607s) [upload=177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:02:56.462815121 (11m12.243748499s) [upload=177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:02:56.495558533 (11m12.276492161s) [seg_pos={0 0}, upload=177]: Begin upload segment... Jul 26 22:02:56.495614233 (11m12.276547669s) [seg_pos={0 0}, upload=177]: Done begin upload segment. Jul 26 22:02:56.495647196 (11m12.276580582s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.495884190 (11m12.276817582s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496275228 (11m12.277208617s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496408089 (11m12.277341472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496668774 (11m12.277602166s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.496871365 (11m12.277804743s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497026953 (11m12.277960343s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497346995 (11m12.278280376s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497445130 (11m12.27837852s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497840356 (11m12.278773746s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.497848583 (11m12.278781967s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498256649 (11m12.279190031s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498328617 (11m12.279262005s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498649087 (11m12.279582472s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.498773449 (11m12.279706817s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.499220962 (11m12.280154343s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Uploading piece... Jul 26 22:02:56.605981739 (11m12.386915116s) [upload=177]: Next returned nil segment Jul 26 22:02:56.606001640 (11m12.386935025s) [upload=177]: Waiting for error group managing segments... Jul 26 22:02:56.610808323 (11m12.391741715s) [noise=true , address=45.91.116.13:4154 , piece_id=6DHPWRS6UL3CKDUK6U3HV4JKJDVQG2TC72WYLZVXOGCTX3TGYG4Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.611449666 (11m12.392383043s) [noise=true , address=45.91.116.6:4973 , piece_id=X5OS7JQBLA7WM6LAAD3G2SBP2FDHKZKT6M3NGZ42OPVVIXVCVCYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612914089 (11m12.393847472s) [noise=true , address=45.91.116.5:4298 , piece_id=4QJ3VV2QKM36YAJJ2O37JOP5CXXIHFY7U3HASTPBWIYGDPPR7IEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.612977922 (11m12.393911289s) [noise=true , address=45.91.116.14:4189 , piece_id=Z4ZYZEFP7YRGFTXR35UWOASLSMCCNA7FFTAIPI4OKFDKAOFFZ6UA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613285867 (11m12.394219244s) [noise=true , address=45.91.116.11:4119 , piece_id=FNSV2RJRBFV6LYXQMZ4VKHZ2EWH7HSMKXGODPTDTEDVHQECSFAIQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613567443 (11m12.394500816s) [noise=true , address=45.91.116.11:4118 , piece_id=KMMC7OFUXGLNSOLQECBHVMAHIRHHBRKUS2CBM6G4CJMODBEOTKAQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.613877717 (11m12.394811096s) [noise=true , address=45.91.116.4:4939 , piece_id=EL2SGB43XY3YXENPP457HD4UMBI6UTMLHAEF7SP3K5YO7WVVO5YA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614080230 (11m12.3950136s) [noise=true , address=45.91.116.7:4010 , piece_id=Z6IVJQ6PGPDXRVBQHTN7EJST32WUNM76WLUN57HWGWUZIUBU3W5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614645429 (11m12.39557882s) [noise=true , address=45.91.116.9:4261 , piece_id=3U4L5EL5EGF3UAN67HVNQ4LTXOAHOBFNY4ODTYVBDVWT5COWGXMQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:02:56.614687861 (11m12.395621241s) [noise=true , address=45.91.116.10:4081 , piece_id=5LV2HWRB2H7VPR3OWLVF7XLTIHJIDVKOOM4TQY6TH7D2MFBZXSEA, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:05.434706200 (11m21.215639563s) [noise=true , address=77.90.31.17:4772 , piece_id=AFBFCY3XRO4D72CDE4UMHMFXUBPXI3BE3S44YKE2B52T6BSPITOQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: <nil> Jul 26 22:03:26.432313509 (11m42.213246898s) [noise=true , address=104.36.82.6:4873 , piece_id=OB3ZPM25W46PWDUBRZBDKSVGKVQ4YOCYDPHFZA65QWQCFPA527FQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:34166->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432428401 (11m42.213361789s) [noise=true , address=104.36.82.4:4801 , piece_id=BMNWG5EAVNHZ3SAJDV76HTII7GH54YOYDADG4JZA4LNEJG3HKX5Q, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:41320->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432451409 (11m42.213384789s) [noise=true , address=104.36.82.7:4912 , piece_id=CRN3WY5KW6C42AE5T2ZDJIQSLASSAHL7AD7SC7WWYNT3ACOBLO6A, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:36250->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432528838 (11m42.213462205s) [noise=true , address=104.36.82.2:4729 , piece_id=2UT7GYTL3SU7FW2SRVTSRBOFORRSGPFZKKSYCP2SZMQAYITP4DYQ, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:50228->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432553381 (11m42.213486767s) [noise=true , address=104.36.82.10:4079 , piece_id=NYTSECG4HTKMQHNVFOH5BDTJXAW4GBPX3MCYGK4UGCOAJZEHORNA, seg_pos={0 0}, upload=177]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:38664->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:26.432601674 (11m42.213535062s) [seg_pos={0 0}, upload=177]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:03:26.464914041 (11m42.245847422s) [upload=177]: Done uploading segments... Jul 26 22:03:26.464946132 (11m42.245879512s) [upload=177]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035716296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XX..XXX..X..X] duration: 30.024375759s FAIL (2025-07-26 22:02:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== op; piecestore: piecestore close: conn closed: op Jul 26 22:02:01.864277882 (10m17.645211265s) [noise=true , address=104.36.82.14:4223 , piece_id=TWHMQMNEEXG73CNQDG4FR6I7CYE6JNWCCGAHOBJUUCLKZ5QYQZYA, seg_pos={0 0}, upload=158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 159.223.143.39:42354->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:01.864297933 (10m17.645231302s) [seg_pos={0 0}, upload=158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:02:01.864436021 (10m17.6453694s) [upload=158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:02:01.864442610 (10m17.645375989s) [upload=158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:02:01.923382249 (10m17.704315631s) [upload=158]: Done uploading segments... Jul 26 22:02:01.923407603 (10m17.704340984s) [upload=158]: Upload finished. err: <nil> Jul 26 22:02:23.258171518 (10m39.039104896s) [upload=167]: Starting upload Jul 26 22:02:23.258277010 (10m39.039210377s) [upload=167]: Done starting upload Jul 26 22:02:23.258438947 (10m39.039372337s) [upload=167]: Uploading segments... Jul 26 22:02:23.258799891 (10m39.039733276s) [upload=167]: Got next segment. Inline: false Jul 26 22:02:23.258861738 (10m39.03979512s) [upload=167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:02:23.258868464 (10m39.039801831s) [upload=167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:02:23.287541782 (10m39.068475168s) [seg_pos={0 0}, upload=167]: Begin upload segment... Jul 26 22:02:23.287619988 (10m39.068553365s) [seg_pos={0 0}, upload=167]: Done begin upload segment. Jul 26 22:02:23.287670639 (10m39.068604024s) [noise=true , address=104.36.82.14:4221 , piece_id=S5JBNYPNQDQ4OK6RTLIDL6GAIH76AUUPZXW6RBFOCV632BMT22YQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.287687980 (10m39.068621364s) [noise=true , address=45.91.116.7:4011 , piece_id=FDDWGMEHHTW5ELDGCKJPCKMBMEEHTKWP7U64255ONMAAEST53LAA, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.288311676 (10m39.069245063s) [noise=true , address=45.91.116.11:4118 , piece_id=XO4UZJE6Y2I6XDXFGHBPPG6JPULZRBF4SBDWIJ6TVBTDY6WGBQWQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.288394511 (10m39.069327907s) [noise=true , address=104.36.82.12:4149 , piece_id=7EDRU57Q54BKALSELJO4HWQLIKSLEZNVYGYAP4ZWCHQKRQNHFJWQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.288745735 (10m39.069679116s) [noise=true , address=45.91.116.6:4974 , piece_id=IW7R6LQNPO2BHI6NB4XYMKVFI6LGIJQFURJR2URPBBIQ7SNUGIOQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.288901884 (10m39.06983527s) [noise=true , address=45.91.116.14:4190 , piece_id=KZ7FWPUN2FCAK74E6C5YW5ZAXUT5EDSVYCVKF6VCWBK4OFMIXQDQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.289083219 (10m39.070016598s) [noise=true , address=45.91.116.11:4119 , piece_id=S5GM4DYJVVW67NX64T2IEIXLWCS6WWGPISP3BIRLUNXEVTRTNI3Q, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.289385089 (10m39.070318479s) [noise=true , address=104.36.82.6:4874 , piece_id=SN4VQBEGUOPLFC5GCZAVBGTWVBCWJ5NHAI4AZH6C3JMPKRPLZEBA, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.289422429 (10m39.070355817s) [noise=true , address=45.91.116.4:4937 , piece_id=WWLPDLAIIK3WWOS5Z5F6JIJAO6IBMHDMEG4AM6SPFK7OOQRVK2EA, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.289778881 (10m39.070712278s) [noise=true , address=45.91.116.12:4225 , piece_id=R3BCXHDZQO6FCBBJHKS6T5PDN4XHUV7CLUC3HL22RBL4ZVLN45RQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.289882833 (10m39.070816216s) [noise=true , address=45.91.116.13:4153 , piece_id=T2J7WKHPQYQC7HQKFNAQS7JMIOVZ7VMCK2SHXHI6KB46ZEIT3OHQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.291145083 (10m39.072078463s) [noise=true , address=45.91.116.8:4048 , piece_id=NP3T423VOTBK2C4ZJQUZGPOTFQUIXCSIC7VISUII534WDK7TQZBA, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.291539127 (10m39.072472513s) [noise=true , address=45.91.116.13:4154 , piece_id=4K24EDA4ZGBWFEB3FPNMX2HXSWUN2IYAEO425JFONAQ4TAMQBTNA, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.292031157 (10m39.072964546s) [noise=true , address=104.36.82.8:4971 , piece_id=ZOZ42EQ6X22YRSIR5OI45YR3Q3G6K72UIYLW35J7YNNCFAGIOO3Q, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.293827982 (10m39.074761359s) [noise=true , address=77.90.31.16:4733 , piece_id=AXIRY7DVGIWICEWJNF3CCZU5YTJCMKGALU47CHYSEMHRB4GUUICQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.296262677 (10m39.077196049s) [noise=true , address=45.91.116.8:4045 , piece_id=A7QTLIYEQKDUP5ERRRY7YK2YPZNU3VOBGXY2PMRGEWFVRJEL4FRQ, seg_pos={0 0}, upload=167]: Uploading piece... Jul 26 22:02:23.388153027 (10m39.169086404s) [upload=167]: Next returned nil segment Jul 26 22:02:23.388175611 (10m39.169109025s) [upload=167]: Waiting for error group managing segments... Jul 26 22:02:23.392428901 (10m39.173362271s) [noise=true , address=45.91.116.13:4154 , piece_id=4K24EDA4ZGBWFEB3FPNMX2HXSWUN2IYAEO425JFONAQ4TAMQBTNA, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.392715547 (10m39.173648941s) [noise=true , address=45.91.116.4:4937 , piece_id=WWLPDLAIIK3WWOS5Z5F6JIJAO6IBMHDMEG4AM6SPFK7OOQRVK2EA, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.392927990 (10m39.17386136s) [noise=true , address=45.91.116.13:4153 , piece_id=T2J7WKHPQYQC7HQKFNAQS7JMIOVZ7VMCK2SHXHI6KB46ZEIT3OHQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.393372602 (10m39.174305981s) [noise=true , address=45.91.116.11:4119 , piece_id=S5GM4DYJVVW67NX64T2IEIXLWCS6WWGPISP3BIRLUNXEVTRTNI3Q, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.393405743 (10m39.174339126s) [noise=true , address=45.91.116.8:4048 , piece_id=NP3T423VOTBK2C4ZJQUZGPOTFQUIXCSIC7VISUII534WDK7TQZBA, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.393789433 (10m39.174722799s) [noise=true , address=45.91.116.11:4118 , piece_id=XO4UZJE6Y2I6XDXFGHBPPG6JPULZRBF4SBDWIJ6TVBTDY6WGBQWQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.393967819 (10m39.174901182s) [noise=true , address=45.91.116.12:4225 , piece_id=R3BCXHDZQO6FCBBJHKS6T5PDN4XHUV7CLUC3HL22RBL4ZVLN45RQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.394252281 (10m39.175185655s) [noise=true , address=45.91.116.6:4974 , piece_id=IW7R6LQNPO2BHI6NB4XYMKVFI6LGIJQFURJR2URPBBIQ7SNUGIOQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.394542882 (10m39.175476264s) [noise=true , address=45.91.116.8:4045 , piece_id=A7QTLIYEQKDUP5ERRRY7YK2YPZNU3VOBGXY2PMRGEWFVRJEL4FRQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.394706141 (10m39.175639503s) [noise=true , address=45.91.116.7:4011 , piece_id=FDDWGMEHHTW5ELDGCKJPCKMBMEEHTKWP7U64255ONMAAEST53LAA, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:23.405250746 (10m39.186184124s) [noise=true , address=45.91.116.14:4190 , piece_id=KZ7FWPUN2FCAK74E6C5YW5ZAXUT5EDSVYCVKF6VCWBK4OFMIXQDQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:27.334904289 (10m43.115837666s) [noise=true , address=77.90.31.16:4733 , piece_id=AXIRY7DVGIWICEWJNF3CCZU5YTJCMKGALU47CHYSEMHRB4GUUICQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: <nil> Jul 26 22:02:53.224603709 (11m9.005537097s) [noise=true , address=104.36.82.14:4221 , piece_id=S5JBNYPNQDQ4OK6RTLIDL6GAIH76AUUPZXW6RBFOCV632BMT22YQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:47670->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.224811594 (11m9.005744973s) [noise=true , address=104.36.82.12:4149 , piece_id=7EDRU57Q54BKALSELJO4HWQLIKSLEZNVYGYAP4ZWCHQKRQNHFJWQ, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:43842->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.224864398 (11m9.005797775s) [noise=true , address=104.36.82.8:4971 , piece_id=ZOZ42EQ6X22YRSIR5OI45YR3Q3G6K72UIYLW35J7YNNCFAGIOO3Q, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.224950351 (11m9.005883738s) [noise=true , address=104.36.82.6:4874 , piece_id=SN4VQBEGUOPLFC5GCZAVBGTWVBCWJ5NHAI4AZH6C3JMPKRPLZEBA, seg_pos={0 0}, upload=167]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:56750->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:53.225017349 (11m9.005950734s) [seg_pos={0 0}, upload=167]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:02:53.247189426 (11m9.028122802s) [upload=167]: Done uploading segments... Jul 26 22:02:53.247221529 (11m9.028154905s) [upload=167]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.024404935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X..X........X] duration: 30.054589877s FAIL (2025-07-26 22:02:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.054621765s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.043190615s FAIL (2025-07-26 22:02:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== ... Jul 26 22:01:31.846468260 (9m47.62740164s) [noise=true , address=64.49.13.140:4508 , piece_id=MGJXP4NX5POOZZT5RTX5VKLHZI27VYTABKUDGLNXUKAGOTWMKVFA, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.846760842 (9m47.62769421s) [noise=true , address=104.36.82.13:4201 , piece_id=LOWLY2IVIOOTTXIFTAVS24YH72UNGGHY7MDVRCEFUUMXG44PWDEA, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.847144172 (9m47.628077554s) [noise=true , address=104.36.82.39:4760 , piece_id=LNH3YD3TY5JWUU4MZGA5I2NZ3SD2RNOSQ2EOKYPBPB4LJWKWT5GQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.847545691 (9m47.628479087s) [noise=true , address=104.36.82.56:4261 , piece_id=H7N5NAANFB2BP3RX24N2V3DGYZI5SASXHQOWIQG74SXHC7YHI4LQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.847613599 (9m47.628546984s) [noise=true , address=104.36.82.11:4138 , piece_id=DAA5M726QPQXDCRFT5PI3OJ2UPMBBM7UIOJ6LSZMCXB2B77TVUIA, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.847945788 (9m47.628879176s) [noise=true , address=79.127.219.38:4007 , piece_id=XQWJFCIKWVI7646ADVNC7BMAVRYDOLRP7MK5237XAVFAXM25QY5A, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.848526995 (9m47.629460381s) [noise=true , address=79.127.220.98:4402 , piece_id=IXF3HFROBVEHI6QZTYIWGN4Q5CUACYXHJWY6GIO7UTUAX5HTJZ2Q, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.848971166 (9m47.629904552s) [noise=true , address=104.36.82.53:4155 , piece_id=SZADNV6XSCP4IQ2KTFHSXIOFTUVSGM6G27A7NDXK6H7IAUGNOIRQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.849405635 (9m47.63033902s) [noise=true , address=104.36.82.46:4496 , piece_id=BJDR77VJO2WXIZDBDPPLVBAKCI6JPMDGWL73XSZPBKG7STKQRVUQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.849795893 (9m47.630729278s) [noise=true , address=104.36.82.4:4829 , piece_id=C44UN6YLZUZXSYUI2ILLJ56IXN4QLOPUKQWF75XA5T66VATADA5Q, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.850238453 (9m47.631171843s) [noise=true , address=77.90.31.10:4536 , piece_id=IRSSIMPSTF3ZCXLCLEEC35LHVHNAGQQ2BSFLFJVOBRN54NCUDVOQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.850675555 (9m47.63160894s) [noise=true , address=79.127.205.235:4676 , piece_id=47T4ZR4NVJBONOON7E7TDDHBSTUNVVSZDAPEGH3CEYIUO5EHMKZQ, seg_pos={0 0}, upload=157]: Uploading piece... Jul 26 22:01:31.945297793 (9m47.726231163s) [upload=157]: Next returned nil segment Jul 26 22:01:31.945320477 (9m47.726253865s) [upload=157]: Waiting for error group managing segments... Jul 26 22:01:31.985259021 (9m47.766192414s) [noise=true , address=79.127.219.38:4007 , piece_id=XQWJFCIKWVI7646ADVNC7BMAVRYDOLRP7MK5237XAVFAXM25QY5A, seg_pos={0 0}, upload=157]: Done uploading piece. err: <nil> Jul 26 22:01:32.422817987 (9m48.203751348s) [noise=true , address=79.127.205.235:4676 , piece_id=47T4ZR4NVJBONOON7E7TDDHBSTUNVVSZDAPEGH3CEYIUO5EHMKZQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: <nil> Jul 26 22:01:35.625074396 (9m51.406007771s) [noise=true , address=45.91.116.14:4206 , piece_id=IFI2SGPSWBRBMMOEKV7VBTU65MFEEZRTQS7DGVXIZCZZIVTGUPRA, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:01:36.346632642 (9m52.127566008s) [noise=true , address=77.90.31.10:4536 , piece_id=IRSSIMPSTF3ZCXLCLEEC35LHVHNAGQQ2BSFLFJVOBRN54NCUDVOQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: <nil> Jul 26 22:01:36.368022512 (9m52.148955881s) [noise=true , address=64.49.13.140:4508 , piece_id=MGJXP4NX5POOZZT5RTX5VKLHZI27VYTABKUDGLNXUKAGOTWMKVFA, seg_pos={0 0}, upload=157]: Done uploading piece. err: <nil> Jul 26 22:01:46.156695676 (10m1.93762904s) [noise=true , address=79.127.220.98:4402 , piece_id=IXF3HFROBVEHI6QZTYIWGN4Q5CUACYXHJWY6GIO7UTUAX5HTJZ2Q, seg_pos={0 0}, upload=157]: Done uploading piece. err: <nil> Jul 26 22:02:31.777233288 (10m47.558166672s) [noise=true , address=104.36.82.7:4942 , piece_id=R6FIGVNFMEMO3ST4EDFGZCHBRALOG7CNPJHB5GFYYP74MR3R22MQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:4942): protocol: write tcp 159.223.143.39:33642->104.36.82.7:4942: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.777473021 (10m47.558406399s) [noise=true , address=104.36.82.55:4221 , piece_id=P4SNA2HOO4643CRZ4JN6HFAWPX2BFAYHEMFY4R46YR3PP6ISFJFQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: write tcp 159.223.143.39:32810->104.36.82.55:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.778565839 (10m47.559499218s) [noise=true , address=104.36.82.52:4090 , piece_id=34OJGRTZR5OLTOFXQJ7HURURE5GWFSK2VKTIGEZXZPF67FQJTO7Q, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 159.223.143.39:43628->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.778676210 (10m47.559609591s) [noise=true , address=104.36.82.13:4201 , piece_id=LOWLY2IVIOOTTXIFTAVS24YH72UNGGHY7MDVRCEFUUMXG44PWDEA, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: write tcp 159.223.143.39:49110->104.36.82.13:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.778766976 (10m47.559714761s) [noise=true , address=104.36.82.39:4760 , piece_id=LNH3YD3TY5JWUU4MZGA5I2NZ3SD2RNOSQ2EOKYPBPB4LJWKWT5GQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:1jJcppBK9zJzGedgY9KEjEA8twqtkdZcjhcnXJoN5AWi5QzR4E, address:104.36.82.39:4760): protocol: write tcp 159.223.143.39:56956->104.36.82.39:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.778870762 (10m47.559804141s) [noise=true , address=104.36.82.56:4261 , piece_id=H7N5NAANFB2BP3RX24N2V3DGYZI5SASXHQOWIQG74SXHC7YHI4LQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: write tcp 159.223.143.39:54876->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.778953120 (10m47.559886498s) [noise=true , address=104.36.82.47:4535 , piece_id=YCLYSAZMSGY524J7RFYQ7STGEY7ZT6ERFGRR2LCAZJCX53CVPONA, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: write tcp 159.223.143.39:43010->104.36.82.47:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.779053181 (10m47.559986566s) [noise=true , address=104.36.82.53:4155 , piece_id=SZADNV6XSCP4IQ2KTFHSXIOFTUVSGM6G27A7NDXK6H7IAUGNOIRQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: write tcp 159.223.143.39:54278->104.36.82.53:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.779901238 (10m47.560834617s) [noise=true , address=104.36.82.12:4177 , piece_id=PMFQBRZEMZR7B2ENRB3TW2ODIJWWS2TPV4RXHB23ST3C6CDIVSEA, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:104.36.82.12:4177): protocol: write tcp 159.223.143.39:51936->104.36.82.12:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.780001169 (10m47.560934548s) [noise=true , address=104.36.82.46:4496 , piece_id=BJDR77VJO2WXIZDBDPPLVBAKCI6JPMDGWL73XSZPBKG7STKQRVUQ, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:54634->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.780123707 (10m47.561057072s) [noise=true , address=104.36.82.4:4829 , piece_id=C44UN6YLZUZXSYUI2ILLJ56IXN4QLOPUKQWF75XA5T66VATADA5Q, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:12aExoQdioFcK1qB46HjSQA6jany7ZxxgRYbBcBu75duXBEKWbi, address:104.36.82.4:4829): protocol: write tcp 159.223.143.39:55766->104.36.82.4:4829: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:31.782059477 (10m47.562992848s) [noise=true , address=104.36.82.11:4138 , piece_id=DAA5M726QPQXDCRFT5PI3OJ2UPMBBM7UIOJ6LSZMCXB2B77TVUIA, seg_pos={0 0}, upload=157]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:02:31.782195835 (10m47.563129221s) [seg_pos={0 0}, upload=157]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:02:31.814520167 (10m47.595453536s) [upload=157]: Done uploading segments... Jul 26 22:02:31.814543078 (10m47.595476455s) [upload=157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.043218713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX..XXX..X..] duration: 16.410976478s FAIL (2025-07-26 22:02:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 2.989691585 (8m28.770625004s) [noise=true , address=104.36.82.2:4731 , piece_id=QRPJKMEUPZ3LHJGFJ5S6T3G5NG2ENN7Z4ZJ4RQR63CDUXOVEC3UA, seg_pos={0 0}, upload=131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 104.36.82.2:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:12.989751767 (8m28.770685153s) [seg_pos={0 0}, upload=131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:00:12.989946654 (8m28.770880031s) [upload=131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:00:12.989955395 (8m28.770888773s) [upload=131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:00:13.038358953 (8m28.819292334s) [upload=131]: Done uploading segments... Jul 26 22:00:13.038397153 (8m28.819330531s) [upload=131]: Upload finished. err: <nil> Jul 26 22:00:36.124445625 (8m51.905378989s) [upload=139]: Starting upload Jul 26 22:00:36.124540841 (8m51.905474229s) [upload=139]: Done starting upload Jul 26 22:00:36.124627574 (8m51.905560961s) [upload=139]: Uploading segments... Jul 26 22:00:36.124876199 (8m51.905809567s) [upload=139]: Got next segment. Inline: false Jul 26 22:00:36.124945039 (8m51.905878418s) [upload=139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:00:36.124951889 (8m51.905885271s) [upload=139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:00:36.154237713 (8m51.935171095s) [seg_pos={0 0}, upload=139]: Begin upload segment... Jul 26 22:00:36.154283827 (8m51.935217191s) [seg_pos={0 0}, upload=139]: Done begin upload segment. Jul 26 22:00:36.154308812 (8m51.93524219s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154352896 (8m51.93528631s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154730497 (8m51.935663882s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154950105 (8m51.935883491s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155133130 (8m51.936066508s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155478387 (8m51.936411775s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.156575134 (8m51.937508499s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.157504923 (8m51.938438304s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.158772840 (8m51.939706249s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.159850367 (8m51.940783751s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.160603605 (8m51.94153699s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161231676 (8m51.942165062s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161978273 (8m51.942911657s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.162939445 (8m51.943872826s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163420866 (8m51.944354241s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163990473 (8m51.944923855s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.297782284 (8m52.078715656s) [upload=139]: Next returned nil segment Jul 26 22:00:36.297804588 (8m52.078737977s) [upload=139]: Waiting for error group managing segments... Jul 26 22:00:36.303949772 (8m52.084883137s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.304670189 (8m52.085603556s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305554607 (8m52.086487973s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305608341 (8m52.086541709s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306253662 (8m52.087187025s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306565464 (8m52.087498834s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306810971 (8m52.087744334s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.308421793 (8m52.089355167s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.309534816 (8m52.090468196s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.310359220 (8m52.091292587s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:51.953721246 (9m7.734654629s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:01:06.091011858 (9m21.871945238s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091175776 (9m21.872109153s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091230499 (9m21.872163877s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091274945 (9m21.872208323s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091304787 (9m21.872238166s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:41188->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091364407 (9m21.872297797s) [seg_pos={0 0}, upload=139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:01:06.120319066 (9m21.901252444s) [upload=139]: Done uploading segments... Jul 26 22:01:06.120351019 (9m21.901284404s) [upload=139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.411016804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XX..XXX..X.] duration: 13.047751188s FAIL (2025-07-26 22:01:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 2.989691585 (8m28.770625004s) [noise=true , address=104.36.82.2:4731 , piece_id=QRPJKMEUPZ3LHJGFJ5S6T3G5NG2ENN7Z4ZJ4RQR63CDUXOVEC3UA, seg_pos={0 0}, upload=131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 104.36.82.2:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:12.989751767 (8m28.770685153s) [seg_pos={0 0}, upload=131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:00:12.989946654 (8m28.770880031s) [upload=131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:00:12.989955395 (8m28.770888773s) [upload=131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:00:13.038358953 (8m28.819292334s) [upload=131]: Done uploading segments... Jul 26 22:00:13.038397153 (8m28.819330531s) [upload=131]: Upload finished. err: <nil> Jul 26 22:00:36.124445625 (8m51.905378989s) [upload=139]: Starting upload Jul 26 22:00:36.124540841 (8m51.905474229s) [upload=139]: Done starting upload Jul 26 22:00:36.124627574 (8m51.905560961s) [upload=139]: Uploading segments... Jul 26 22:00:36.124876199 (8m51.905809567s) [upload=139]: Got next segment. Inline: false Jul 26 22:00:36.124945039 (8m51.905878418s) [upload=139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:00:36.124951889 (8m51.905885271s) [upload=139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:00:36.154237713 (8m51.935171095s) [seg_pos={0 0}, upload=139]: Begin upload segment... Jul 26 22:00:36.154283827 (8m51.935217191s) [seg_pos={0 0}, upload=139]: Done begin upload segment. Jul 26 22:00:36.154308812 (8m51.93524219s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154352896 (8m51.93528631s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154730497 (8m51.935663882s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154950105 (8m51.935883491s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155133130 (8m51.936066508s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155478387 (8m51.936411775s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.156575134 (8m51.937508499s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.157504923 (8m51.938438304s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.158772840 (8m51.939706249s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.159850367 (8m51.940783751s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.160603605 (8m51.94153699s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161231676 (8m51.942165062s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161978273 (8m51.942911657s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.162939445 (8m51.943872826s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163420866 (8m51.944354241s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163990473 (8m51.944923855s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.297782284 (8m52.078715656s) [upload=139]: Next returned nil segment Jul 26 22:00:36.297804588 (8m52.078737977s) [upload=139]: Waiting for error group managing segments... Jul 26 22:00:36.303949772 (8m52.084883137s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.304670189 (8m52.085603556s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305554607 (8m52.086487973s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305608341 (8m52.086541709s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306253662 (8m52.087187025s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306565464 (8m52.087498834s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306810971 (8m52.087744334s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.308421793 (8m52.089355167s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.309534816 (8m52.090468196s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.310359220 (8m52.091292587s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:51.953721246 (9m7.734654629s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:01:06.091011858 (9m21.871945238s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091175776 (9m21.872109153s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091230499 (9m21.872163877s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091274945 (9m21.872208323s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091304787 (9m21.872238166s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:41188->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091364407 (9m21.872297797s) [seg_pos={0 0}, upload=139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:01:06.120319066 (9m21.901252444s) [upload=139]: Done uploading segments... Jul 26 22:01:06.120351019 (9m21.901284404s) [upload=139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 13.047785965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.030158424s FAIL (2025-07-26 22:01:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 054254s) [noise=true , address=104.36.82.14:4232 , piece_id=4IONOBYVI7AXVYGDV7YAYUG5H7QKOXN7BNLMBBGFK75K6NZIFDOA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.747554317 (8m47.52848769s) [noise=true , address=104.36.82.55:4194 , piece_id=K7F67VFEGOAXTP7YPL2DHP73PCCLIZRM23HGWQLSIWEJ5QRF75RA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.747972433 (8m47.528905826s) [noise=true , address=104.36.82.54:4176 , piece_id=7763YFPJ7SUFEZQHWADXQYUNAZBPIGHN4FCIDKA2I4Y7ZUKPQWXA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.748525240 (8m47.529458635s) [noise=true , address=104.36.82.37:4679 , piece_id=EFG5LF4PZCSYBPJ72HA43YX2X3KXE4JMFVIEZSQTIXDTFM4DEMVQ, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.748954478 (8m47.529887866s) [noise=true , address=79.127.205.246:4286 , piece_id=QDNAAGOV64DTLME2XOM6J52TD3FLMNG4PAESVSJVDC75EA6WLK5Q, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.749382666 (8m47.530316055s) [noise=true , address=156.146.43.225:4449 , piece_id=OPMU6MOGU2BF5J47BNPXCGKU4IO5UPSPD2O5DASIQEY42SB453KA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.749808273 (8m47.530741664s) [noise=true , address=79.127.205.245:4244 , piece_id=7R33DZH7QHPNSBS5P452H636OHJ6DP4WRBVQLVXGN26PSG72JTWA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.750206247 (8m47.531139636s) [noise=true , address=104.36.82.47:4544 , piece_id=B67LIYCWAV3A25YUXJGUFVOURHUKISIIOZ3SW3HMULBPBYNBPB4Q, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.750606345 (8m47.531539729s) [noise=true , address=104.36.82.46:4504 , piece_id=PU6LZQHQHXEATNRNE3MW4VZO2OS2QTDUXIQGJ2CHEX2YZW3OC35A, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.751001051 (8m47.531934441s) [noise=true , address=79.127.220.98:4402 , piece_id=LRKXRTQJGOW32LHOSSNFRLSHO6CAS2LDYBSSNS3JRRKZBGETFQ2A, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.751344520 (8m47.53227791s) [noise=true , address=77.90.31.16:4755 , piece_id=WAYTBQWOX7SP2MEVFUHS4E35CBZTY5FZGCUR75URIN3TVK6KSNEQ, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.751790467 (8m47.53272386s) [noise=true , address=104.36.82.52:4094 , piece_id=GDMW7EZLPFQCBVZQAPT5GSIMOBCPGMBBO3K6QXWNE4WRZAAF557A, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.753517223 (8m47.534450613s) [noise=true , address=79.127.201.215:4345 , piece_id=NMIQT3JCQ7ZLFOKMCHSYXZETE242367RNYJSAV4LWR6IVEOKGM4Q, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.754015733 (8m47.534949117s) [noise=true , address=104.36.82.44:4877 , piece_id=EEYDXEB4SEP6YMELEFVWWRV5A663O67ZU6VI7OIQT5DHJ7J24CHA, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.754622672 (8m47.535556059s) [noise=true , address=104.36.82.6:4906 , piece_id=CTEGS62V4Y3G3R5BBNU6WASC6DMAVD56HH4IBH5KT5V3AMGPHWPQ, seg_pos={0 0}, upload=138]: Uploading piece... Jul 26 22:00:31.845352107 (8m47.626285492s) [upload=138]: Next returned nil segment Jul 26 22:00:31.845375637 (8m47.626309018s) [upload=138]: Waiting for error group managing segments... Jul 26 22:00:32.005962321 (8m47.786895699s) [noise=true , address=156.146.43.225:4449 , piece_id=OPMU6MOGU2BF5J47BNPXCGKU4IO5UPSPD2O5DASIQEY42SB453KA, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:32.027677446 (8m47.808610823s) [noise=true , address=79.127.220.98:4402 , piece_id=LRKXRTQJGOW32LHOSSNFRLSHO6CAS2LDYBSSNS3JRRKZBGETFQ2A, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:32.316130356 (8m48.097063739s) [noise=true , address=79.127.205.245:4244 , piece_id=7R33DZH7QHPNSBS5P452H636OHJ6DP4WRBVQLVXGN26PSG72JTWA, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:32.414272704 (8m48.195206097s) [noise=true , address=79.127.205.246:4286 , piece_id=QDNAAGOV64DTLME2XOM6J52TD3FLMNG4PAESVSJVDC75EA6WLK5Q, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:34.036576295 (8m49.817509665s) [noise=true , address=79.127.219.47:4069 , piece_id=2M4T2XWBRTMNKMVPDNVXFYJES3UYMIMHRLYV67SFVP5TKZAOI7MA, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:37.045448412 (8m52.826381796s) [noise=true , address=77.90.31.16:4755 , piece_id=WAYTBQWOX7SP2MEVFUHS4E35CBZTY5FZGCUR75URIN3TVK6KSNEQ, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:00:37.284529717 (8m53.065463095s) [noise=true , address=79.127.201.215:4345 , piece_id=NMIQT3JCQ7ZLFOKMCHSYXZETE242367RNYJSAV4LWR6IVEOKGM4Q, seg_pos={0 0}, upload=138]: Done uploading piece. err: <nil> Jul 26 22:01:31.666617286 (9m47.44755067s) [noise=true , address=104.36.82.56:4241 , piece_id=4I4GIMHPJBBMHTNNRSMWELQB5NQ2NW6LLJU2BB4WCINWBRWLVPJQ, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: rpc: dial tcp 104.36.82.56:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.666713780 (9m47.447647158s) [noise=true , address=104.36.82.3:4787 , piece_id=YYRMRBEUQXQUIOU67TDUZXYOWCBRTL5Q7JMVUXJCFSABDGON7HIA, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:104.36.82.3:4787): protocol: write tcp 159.223.143.39:46644->104.36.82.3:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.666789366 (9m47.447722743s) [noise=true , address=104.36.82.14:4232 , piece_id=4IONOBYVI7AXVYGDV7YAYUG5H7QKOXN7BNLMBBGFK75K6NZIFDOA, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1jWewg226E4BrBxqcK38n4ZjHx3toQeNdbc5HeU2EoykztXwyX, address:104.36.82.14:4232): protocol: write tcp 159.223.143.39:39080->104.36.82.14:4232: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.666854099 (9m47.447787464s) [noise=true , address=104.36.82.6:4906 , piece_id=CTEGS62V4Y3G3R5BBNU6WASC6DMAVD56HH4IBH5KT5V3AMGPHWPQ, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1gByS6jTUkLuHrY8MafgmkmB4wXyY2j9Mt48fkpPNWbCKTcnty, address:104.36.82.6:4906): protocol: write tcp 159.223.143.39:37648->104.36.82.6:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.666913929 (9m47.447847308s) [noise=true , address=104.36.82.47:4544 , piece_id=B67LIYCWAV3A25YUXJGUFVOURHUKISIIOZ3SW3HMULBPBYNBPB4Q, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: write tcp 159.223.143.39:43374->104.36.82.47:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.667222299 (9m47.448155682s) [noise=true , address=104.36.82.44:4877 , piece_id=EEYDXEB4SEP6YMELEFVWWRV5A663O67ZU6VI7OIQT5DHJ7J24CHA, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: write tcp 159.223.143.39:45912->104.36.82.44:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.667284071 (9m47.448217454s) [noise=true , address=104.36.82.52:4094 , piece_id=GDMW7EZLPFQCBVZQAPT5GSIMOBCPGMBBO3K6QXWNE4WRZAAF557A, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 159.223.143.39:33968->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.667326226 (9m47.448259609s) [noise=true , address=104.36.82.37:4679 , piece_id=EFG5LF4PZCSYBPJ72HA43YX2X3KXE4JMFVIEZSQTIXDTFM4DEMVQ, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1VHQihNGcx6bTvwitH9PCSqyuKWuchdYhYzZXwSuoruQFByGoj, address:104.36.82.37:4679): protocol: write tcp 159.223.143.39:46798->104.36.82.37:4679: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.667581173 (9m47.448514556s) [noise=true , address=104.36.82.54:4176 , piece_id=7763YFPJ7SUFEZQHWADXQYUNAZBPIGHN4FCIDKA2I4Y7ZUKPQWXA, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.667650189 (9m47.448583566s) [noise=true , address=104.36.82.46:4504 , piece_id=PU6LZQHQHXEATNRNE3MW4VZO2OS2QTDUXIQGJ2CHEX2YZW3OC35A, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:31.668033850 (9m47.448967227s) [noise=true , address=104.36.82.55:4194 , piece_id=K7F67VFEGOAXTP7YPL2DHP73PCCLIZRM23HGWQLSIWEJ5QRF75RA, seg_pos={0 0}, upload=138]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:01:31.668073996 (9m47.449007379s) [seg_pos={0 0}, upload=138]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:01:31.693682234 (9m47.474615613s) [upload=138]: Done uploading segments... Jul 26 22:01:31.693713603 (9m47.474646986s) [upload=138]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.030179981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..X........] duration: 5.192799186s FAIL (2025-07-26 22:01:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.192833431s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX..XXX..X] duration: 30.034196204s FAIL (2025-07-26 22:01:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 2.989691585 (8m28.770625004s) [noise=true , address=104.36.82.2:4731 , piece_id=QRPJKMEUPZ3LHJGFJ5S6T3G5NG2ENN7Z4ZJ4RQR63CDUXOVEC3UA, seg_pos={0 0}, upload=131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 104.36.82.2:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:12.989751767 (8m28.770685153s) [seg_pos={0 0}, upload=131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:00:12.989946654 (8m28.770880031s) [upload=131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:00:12.989955395 (8m28.770888773s) [upload=131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:00:13.038358953 (8m28.819292334s) [upload=131]: Done uploading segments... Jul 26 22:00:13.038397153 (8m28.819330531s) [upload=131]: Upload finished. err: <nil> Jul 26 22:00:36.124445625 (8m51.905378989s) [upload=139]: Starting upload Jul 26 22:00:36.124540841 (8m51.905474229s) [upload=139]: Done starting upload Jul 26 22:00:36.124627574 (8m51.905560961s) [upload=139]: Uploading segments... Jul 26 22:00:36.124876199 (8m51.905809567s) [upload=139]: Got next segment. Inline: false Jul 26 22:00:36.124945039 (8m51.905878418s) [upload=139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:00:36.124951889 (8m51.905885271s) [upload=139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:00:36.154237713 (8m51.935171095s) [seg_pos={0 0}, upload=139]: Begin upload segment... Jul 26 22:00:36.154283827 (8m51.935217191s) [seg_pos={0 0}, upload=139]: Done begin upload segment. Jul 26 22:00:36.154308812 (8m51.93524219s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154352896 (8m51.93528631s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154730497 (8m51.935663882s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.154950105 (8m51.935883491s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155133130 (8m51.936066508s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.155478387 (8m51.936411775s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.156575134 (8m51.937508499s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.157504923 (8m51.938438304s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.158772840 (8m51.939706249s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.159850367 (8m51.940783751s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.160603605 (8m51.94153699s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161231676 (8m51.942165062s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.161978273 (8m51.942911657s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.162939445 (8m51.943872826s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163420866 (8m51.944354241s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.163990473 (8m51.944923855s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Uploading piece... Jul 26 22:00:36.297782284 (8m52.078715656s) [upload=139]: Next returned nil segment Jul 26 22:00:36.297804588 (8m52.078737977s) [upload=139]: Waiting for error group managing segments... Jul 26 22:00:36.303949772 (8m52.084883137s) [noise=true , address=45.91.116.14:4190 , piece_id=DCEXAHW4OLNXUQ6LCC37LC3DBBW66RDTYK7G4DGNUR466LMGXL4Q, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.304670189 (8m52.085603556s) [noise=true , address=45.91.116.11:4118 , piece_id=36N4Z4YEKYCAIYHUMDY5XSVNKY6R5RBTS7TIVETH6EM5CWG6C6UA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305554607 (8m52.086487973s) [noise=true , address=45.91.116.9:4263 , piece_id=TPRTSUIHNESF4OUPNXWCUK5GDZ2BXH7KHJFFTK6OW5O2ZFHEB25A, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.305608341 (8m52.086541709s) [noise=true , address=45.91.116.12:4226 , piece_id=S327XWRJ3GROOXAJ2VZ5TKVK5L2F42PQ4TXRWVVPBBJFYK3NALUA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306253662 (8m52.087187025s) [noise=true , address=45.91.116.9:4261 , piece_id=JLEI4CNF25JYVCQKRXTDZYYVHATVBPSUYO37EJX25JJPHCRGXMLA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306565464 (8m52.087498834s) [noise=true , address=45.91.116.5:4299 , piece_id=PHY5FLL4BHPE4SGCALQ46OW5CCEVCYAXMZPHKVLQIOXNUSSEOZNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.306810971 (8m52.087744334s) [noise=true , address=45.91.116.7:4011 , piece_id=F4DNIT7UU2F7TUXU45EQ4XFUQG2KEAKOFGATMPUWS7DRKNOWPAMQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.308421793 (8m52.089355167s) [noise=true , address=45.91.116.14:4191 , piece_id=VZIHNG3I2AQQAYZH3HMGHVHQ7GCGQZ7I6SH6K46XNXKFLBWVU5VA, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.309534816 (8m52.090468196s) [noise=true , address=45.91.116.8:4047 , piece_id=EI4UUJB7IJBRF6XO3R6HZEMNIXUSA3GX2TXTERQS45XKYOWOTQVQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:36.310359220 (8m52.091292587s) [noise=true , address=45.91.116.13:4155 , piece_id=FAPDCQANSRH6EZI5OWMTSZTUEDHDOQJ5QKTT7HZCVYMQ4QLTHWRQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:00:51.953721246 (9m7.734654629s) [noise=true , address=77.90.31.19:4843 , piece_id=Q6QNBZ2DEPXPTFX5MZT2EHBDPPY7XPLUGK44RL666EPLPVGC24ZQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: <nil> Jul 26 22:01:06.091011858 (9m21.871945238s) [noise=true , address=104.36.82.9:4006 , piece_id=PJ455VKC2AW7B5X7CY7FOFZTL4FHAZUBTHVVQEOV536PQKW5KYQQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091175776 (9m21.872109153s) [noise=true , address=104.36.82.2:4732 , piece_id=EQSCQ26ZXVHJSHDSMQGQBBUXENE23UFJZHSPJRUMKF67SUFF2RPQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091230499 (9m21.872163877s) [noise=true , address=104.36.82.3:4765 , piece_id=CGY44J7THVF5HTBSGZSKMU2M2VCYQ5BLZS4DNEOXYFR5GP7JGJFQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091274945 (9m21.872208323s) [noise=true , address=104.36.82.11:4114 , piece_id=3SJ2AIOWURHUFCTIQNCP4MMX5AEXCD6AVNBIYSBWJ62GQXMA23WA, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091304787 (9m21.872238166s) [noise=true , address=104.36.82.6:4875 , piece_id=5EFQVT2KXHYUJGEFHW7CPASHXXGKIRVWZGJLJ4IUM3GRSDBODBNQ, seg_pos={0 0}, upload=139]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:41188->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:06.091364407 (9m21.872297797s) [seg_pos={0 0}, upload=139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:01:06.120319066 (9m21.901252444s) [upload=139]: Done uploading segments... Jul 26 22:01:06.120351019 (9m21.901284404s) [upload=139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.034229249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration: 3.906910265s FAIL (2025-07-26 22:01:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.906946072s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X..X......] duration: 26.249873152s FAIL (2025-07-26 22:00:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 26.249923508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.026161248s FAIL (2025-07-26 22:00:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== :31.643214629 (7m47.424148013s) [noise=true , address=104.36.82.47:4524 , piece_id=NPUA5WBOQVHIX7OPZKXBZKXCJN27WMLVBXNUP567QMDMXEQZKD6A, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.643260922 (7m47.4241943s) [noise=true , address=104.36.82.46:4491 , piece_id=7PIB3AGU553UBJFOHNJV4RU2RXDP7UZ2VIDYDBLU7OPNHZFL6VZQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.643674529 (7m47.42460791s) [noise=true , address=104.36.82.53:4145 , piece_id=ABR75FULC6YKGSW7EB2DZACCPHMON2P5ZTX4JNMS4PZJ5ILET3NQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.643858435 (7m47.424791823s) [noise=true , address=104.36.82.54:4162 , piece_id=QCOCU3K5QRNLWU4SGR4VRAHWRWFVAVFGYIK2DHF3Z5NYPHAEXUMQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.644556322 (7m47.425489706s) [noise=true , address=104.36.82.50:4043 , piece_id=6ADTJSLASZF72JYMIAOMWIPTV2KMHFBIFBP6CKP2MG7ZX24QXNJQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.646768571 (7m47.427701958s) [noise=true , address=79.127.205.246:4286 , piece_id=GWYIXRGNXVTRUFAQ5KOIFC2GPM3D4EHQOQ5SA5Y7GEIPQR3BXSPA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.647461806 (7m47.42839519s) [noise=true , address=104.36.82.8:4975 , piece_id=G2SE5UEB26YEDGYZBLCBIGMHYZAOV2TGAIQIYUY4IAUW6UGYKRFA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.648037581 (7m47.428970967s) [noise=true , address=79.127.205.234:4601 , piece_id=6Q4Z7ARSHTIVELFLTEPMQMNEXYANFDW2YXV5TSPIOYSE5H5C2OAA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.651232463 (7m47.432165849s) [noise=true , address=77.90.31.10:4531 , piece_id=FQHJV2QPGVR2TO3WRFDIRR7Z3TWTSMGFX423YWPJV4WHGCFZJC5Q, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.651818278 (7m47.432751661s) [noise=true , address=104.36.82.3:4785 , piece_id=BKUFMIEOUFJGIK3BWRVTYQRXRUMU7LU7ACL4NT7NOQG5T44B2QKA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.652477214 (7m47.4334106s) [noise=true , address=64.49.13.154:4935 , piece_id=XC4G5CCHS3HBZ5M5LMTOZGHCBNWS5HE3X7UZ5VYHJAOEZFNDKMFA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.652936789 (7m47.433870171s) [noise=true , address=104.36.82.55:4206 , piece_id=VGVH6KCUNRXKMRIL3K2YNSMNP3Z25RBM7FG46MRQJWRBEYLELUAQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.653392721 (7m47.434326109s) [noise=true , address=104.36.82.51:4078 , piece_id=Q7ETLK6XAQDZOGZ24VQOESYICN7II3URHHOOMU2U2F3F7K45ICKQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.653899630 (7m47.434833015s) [noise=true , address=104.36.82.52:4101 , piece_id=CVBB44K5GCYFNFA657IDYZ5UGZ6ZWBMVNLUCLEKSXVYHHE4KFGMQ, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.654373782 (7m47.435307163s) [noise=true , address=79.127.201.215:4358 , piece_id=4A3WHQX7Z2CUMTWYTJAWY3FVJWCSDQMP4JR5VHP5GRJZMFKGDHAA, seg_pos={0 0}, upload=122]: Uploading piece... Jul 26 21:59:31.717460555 (7m47.498393935s) [upload=122]: Next returned nil segment Jul 26 21:59:31.717481342 (7m47.498414726s) [upload=122]: Waiting for error group managing segments... Jul 26 21:59:31.755412114 (7m47.536345491s) [noise=true , address=79.127.219.38:4004 , piece_id=K27MU5Q2Q4L7ZFGSJKRXFQUEGCG3JVAJ4QQ6UCTCX5D6MERKCL5Q, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:32.252210305 (7m48.033143692s) [noise=true , address=79.127.205.234:4601 , piece_id=6Q4Z7ARSHTIVELFLTEPMQMNEXYANFDW2YXV5TSPIOYSE5H5C2OAA, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:32.316641554 (7m48.097574925s) [noise=true , address=79.127.205.246:4286 , piece_id=GWYIXRGNXVTRUFAQ5KOIFC2GPM3D4EHQOQ5SA5Y7GEIPQR3BXSPA, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:32.881286023 (7m48.662219393s) [noise=true , address=64.49.13.154:4935 , piece_id=XC4G5CCHS3HBZ5M5LMTOZGHCBNWS5HE3X7UZ5VYHJAOEZFNDKMFA, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:32.927244331 (7m48.708177711s) [noise=true , address=79.127.219.47:4061 , piece_id=T7XP4INDUP6PJ2FXOYSQ7W3DZF2DNRBUQYMJBZGOOL3OKAPEPUFQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:33.404098446 (7m49.185031835s) [noise=true , address=79.127.201.215:4358 , piece_id=4A3WHQX7Z2CUMTWYTJAWY3FVJWCSDQMP4JR5VHP5GRJZMFKGDHAA, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 21:59:37.771084754 (7m53.55201813s) [noise=true , address=77.90.31.10:4531 , piece_id=FQHJV2QPGVR2TO3WRFDIRR7Z3TWTSMGFX423YWPJV4WHGCFZJC5Q, seg_pos={0 0}, upload=122]: Done uploading piece. err: <nil> Jul 26 22:00:31.578698176 (8m47.359631561s) [noise=true , address=104.36.82.7:4923 , piece_id=ZQJWWESOPBB747HBLJTOIXDOVYARSBHSSEUONCAXCC4UQOTBDY7A, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:104.36.82.7:4923): protocol: rpc: dial tcp 104.36.82.7:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.578805326 (8m47.359738705s) [noise=true , address=104.36.82.8:4975 , piece_id=G2SE5UEB26YEDGYZBLCBIGMHYZAOV2TGAIQIYUY4IAUW6UGYKRFA, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:104.36.82.8:4975): protocol: rpc: dial tcp 104.36.82.8:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.578880172 (8m47.359813554s) [noise=true , address=104.36.82.47:4524 , piece_id=NPUA5WBOQVHIX7OPZKXBZKXCJN27WMLVBXNUP567QMDMXEQZKD6A, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 159.223.143.39:38524->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579162572 (8m47.360095954s) [noise=true , address=104.36.82.46:4491 , piece_id=7PIB3AGU553UBJFOHNJV4RU2RXDP7UZ2VIDYDBLU7OPNHZFL6VZQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: write tcp 159.223.143.39:44924->104.36.82.46:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579271270 (8m47.360204647s) [noise=true , address=104.36.82.53:4145 , piece_id=ABR75FULC6YKGSW7EB2DZACCPHMON2P5ZTX4JNMS4PZJ5ILET3NQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: rpc: dial tcp 104.36.82.53:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579371106 (8m47.360304486s) [noise=true , address=104.36.82.55:4206 , piece_id=VGVH6KCUNRXKMRIL3K2YNSMNP3Z25RBM7FG46MRQJWRBEYLELUAQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579459914 (8m47.360393293s) [noise=true , address=104.36.82.54:4162 , piece_id=QCOCU3K5QRNLWU4SGR4VRAHWRWFVAVFGYIK2DHF3Z5NYPHAEXUMQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:54302->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579495858 (8m47.360429247s) [noise=true , address=104.36.82.52:4101 , piece_id=CVBB44K5GCYFNFA657IDYZ5UGZ6ZWBMVNLUCLEKSXVYHHE4KFGMQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:36034->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579581978 (8m47.360515358s) [noise=true , address=104.36.82.50:4043 , piece_id=6ADTJSLASZF72JYMIAOMWIPTV2KMHFBIFBP6CKP2MG7ZX24QXNJQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:104.36.82.50:4043): protocol: rpc: dial tcp 104.36.82.50:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579651480 (8m47.360584857s) [noise=true , address=104.36.82.3:4785 , piece_id=BKUFMIEOUFJGIK3BWRVTYQRXRUMU7LU7ACL4NT7NOQG5T44B2QKA, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:104.36.82.3:4785): protocol: write tcp 159.223.143.39:51612->104.36.82.3:4785: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579665433 (8m47.360598801s) [noise=true , address=104.36.82.51:4078 , piece_id=Q7ETLK6XAQDZOGZ24VQOESYICN7II3URHHOOMU2U2F3F7K45ICKQ, seg_pos={0 0}, upload=122]: Done uploading piece. err: ecclient: upload failed (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd, address:104.36.82.51:4078): protocol: rpc: dial tcp 104.36.82.51:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.579721241 (8m47.360654627s) [seg_pos={0 0}, upload=122]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:00:31.602742141 (8m47.383675535s) [upload=122]: Done uploading segments... Jul 26 22:00:31.602774251 (8m47.383707645s) [upload=122]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.026181164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX..XXX..] duration: 7.252263598s FAIL (2025-07-26 22:00:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== PdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802012332 (7m12.582945696s) [noise=true , address=104.36.82.13:4186 , piece_id=JK3XUENYVYRYGIXBI5GEKHO3CTB23XU4OPPOCSVOIL5Y3RYQCMQA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802031194 (7m12.582964575s) [seg_pos={0 0}, upload=100]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:58:56.828924440 (7m12.609857809s) [upload=100]: Done uploading segments... Jul 26 21:58:56.828953351 (7m12.609886727s) [upload=100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:58:56.928383445 (7m12.70931681s) [upload=112]: Starting upload Jul 26 21:58:56.928483411 (7m12.709416796s) [upload=112]: Done starting upload Jul 26 21:58:56.928599078 (7m12.709532445s) [upload=112]: Uploading segments... Jul 26 21:58:56.928846534 (7m12.7097799s) [upload=112]: Got next segment. Inline: false Jul 26 21:58:56.928903063 (7m12.709836429s) [upload=112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:58:56.928909218 (7m12.709842599s) [upload=112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:58:56.962397706 (7m12.743331082s) [seg_pos={0 0}, upload=112]: Begin upload segment... Jul 26 21:58:56.962446320 (7m12.743379685s) [seg_pos={0 0}, upload=112]: Done begin upload segment. Jul 26 21:58:56.962486844 (7m12.743420221s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962504965 (7m12.743438346s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962963964 (7m12.743897353s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962989197 (7m12.743922579s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963409054 (7m12.744342436s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963423183 (7m12.744356568s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963781349 (7m12.744714731s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964537876 (7m12.745471253s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964930612 (7m12.745863994s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.965349574 (7m12.746282953s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.966552138 (7m12.74748552s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967124362 (7m12.74805774s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967724725 (7m12.748658107s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968251177 (7m12.749184569s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968796893 (7m12.749730277s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.969667421 (7m12.75060079s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:57.057172667 (7m12.838106037s) [upload=112]: Next returned nil segment Jul 26 21:58:57.057190660 (7m12.838124044s) [upload=112]: Waiting for error group managing segments... Jul 26 21:58:57.061079888 (7m12.842013253s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.061664155 (7m12.842597519s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062174746 (7m12.843108108s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062588806 (7m12.843522176s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063003156 (7m12.84393652s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063537160 (7m12.844470526s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063593642 (7m12.844527007s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063923281 (7m12.844856646s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.064066458 (7m12.844999823s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:00.371963705 (7m16.15289707s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:05.374869080 (7m21.155802458s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:26.900101378 (7m42.681034761s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:54702->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900206538 (7m42.681139914s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:56252->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900260676 (7m42.681194052s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:43016->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900359586 (7m42.681292969s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900429335 (7m42.681362712s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:55284->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900454028 (7m42.681387411s) [seg_pos={0 0}, upload=112]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:59:26.927954817 (7m42.708888207s) [upload=112]: Done uploading segments... Jul 26 21:59:26.927985802 (7m42.708919184s) [upload=112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.252292097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration: 6.553478597s FAIL (2025-07-26 21:59:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.553510909s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XX..XXX.] duration: 6.6482671s FAIL (2025-07-26 21:59:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== PdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802012332 (7m12.582945696s) [noise=true , address=104.36.82.13:4186 , piece_id=JK3XUENYVYRYGIXBI5GEKHO3CTB23XU4OPPOCSVOIL5Y3RYQCMQA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802031194 (7m12.582964575s) [seg_pos={0 0}, upload=100]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:58:56.828924440 (7m12.609857809s) [upload=100]: Done uploading segments... Jul 26 21:58:56.828953351 (7m12.609886727s) [upload=100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:58:56.928383445 (7m12.70931681s) [upload=112]: Starting upload Jul 26 21:58:56.928483411 (7m12.709416796s) [upload=112]: Done starting upload Jul 26 21:58:56.928599078 (7m12.709532445s) [upload=112]: Uploading segments... Jul 26 21:58:56.928846534 (7m12.7097799s) [upload=112]: Got next segment. Inline: false Jul 26 21:58:56.928903063 (7m12.709836429s) [upload=112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:58:56.928909218 (7m12.709842599s) [upload=112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:58:56.962397706 (7m12.743331082s) [seg_pos={0 0}, upload=112]: Begin upload segment... Jul 26 21:58:56.962446320 (7m12.743379685s) [seg_pos={0 0}, upload=112]: Done begin upload segment. Jul 26 21:58:56.962486844 (7m12.743420221s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962504965 (7m12.743438346s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962963964 (7m12.743897353s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962989197 (7m12.743922579s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963409054 (7m12.744342436s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963423183 (7m12.744356568s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963781349 (7m12.744714731s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964537876 (7m12.745471253s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964930612 (7m12.745863994s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.965349574 (7m12.746282953s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.966552138 (7m12.74748552s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967124362 (7m12.74805774s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967724725 (7m12.748658107s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968251177 (7m12.749184569s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968796893 (7m12.749730277s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.969667421 (7m12.75060079s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:57.057172667 (7m12.838106037s) [upload=112]: Next returned nil segment Jul 26 21:58:57.057190660 (7m12.838124044s) [upload=112]: Waiting for error group managing segments... Jul 26 21:58:57.061079888 (7m12.842013253s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.061664155 (7m12.842597519s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062174746 (7m12.843108108s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062588806 (7m12.843522176s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063003156 (7m12.84393652s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063537160 (7m12.844470526s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063593642 (7m12.844527007s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063923281 (7m12.844856646s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.064066458 (7m12.844999823s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:00.371963705 (7m16.15289707s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:05.374869080 (7m21.155802458s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:26.900101378 (7m42.681034761s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:54702->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900206538 (7m42.681139914s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:56252->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900260676 (7m42.681194052s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:43016->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900359586 (7m42.681292969s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900429335 (7m42.681362712s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:55284->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900454028 (7m42.681387411s) [seg_pos={0 0}, upload=112]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:59:26.927954817 (7m42.708888207s) [upload=112]: Done uploading segments... Jul 26 21:59:26.927985802 (7m42.708919184s) [upload=112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.64829577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.035225904s FAIL (2025-07-26 21:59:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== e=true , address=104.36.82.22:4529 , piece_id=A3O3SVOFVEWV3FKHMFK4C2LJQ3DBZVDQILQKWOZHJV6EU2MCPZFQ, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.539671568 (6m47.320604954s) [noise=true , address=77.90.31.16:4766 , piece_id=4N3DJG7Z7TGBLW4ELOB25SDN65UZUAIU5LEPILOFV3PQHSPYIUEA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.540070793 (6m47.321004191s) [noise=true , address=104.36.82.10:4096 , piece_id=ZM65X7BOHPV556VUDHD6GP5KR37QS6RHEO3U3UZ7UREVBVKQTSJA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.540686488 (6m47.321619874s) [noise=true , address=104.36.82.53:4124 , piece_id=YZBFYCO5UYONAGYAIOWSFTZVQPZT4Z2PVOOL3ARW7JMSWHZMTNNQ, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.541027763 (6m47.32196115s) [noise=true , address=104.36.82.3:4787 , piece_id=PBQK4YYE2ZTB3YJQOIPDC5H6UUXVYPTAFM6BWFKB6JAV36JECADA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.541241916 (6m47.322175293s) [noise=true , address=77.90.31.18:4833 , piece_id=57GAN7M3VDVGGTTSOO3VEVMEALFI7DOKCMBJ6CWTKDBIE2QH3DQA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.541764703 (6m47.322698085s) [noise=true , address=79.127.219.38:4005 , piece_id=NHI7KKCLXHQSSSJ4MEUEPZLVGFE7MW63FZPG7TEMEPKQ57E6TCXA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.542092496 (6m47.323025883s) [noise=true , address=104.36.82.47:4518 , piece_id=IIO4W6SNZNYDRAGQGZGK6UDC2JXCCIJNU7TRO4NPVCEYNXJJ2LZQ, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.542316224 (6m47.32324961s) [noise=true , address=104.36.82.49:4009 , piece_id=EFZCCYNDQKOJ4SO6TVRKCTFO6Z26CTKNRQ6QDLPOWYGHQD4J2NUQ, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.542481083 (6m47.323414468s) [noise=true , address=104.36.82.52:4118 , piece_id=BHLNV2EREN465VRGZGD7BNIHAVYHSQCDSBW7AEZQL2Y7OEGTJUHA, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.543143911 (6m47.324077293s) [noise=true , address=104.36.82.54:4191 , piece_id=CTZBMQF62W7ZTWYCC5LCB43NEIQIVIX7BXUOORM57Z5ZZAD2AZ7A, seg_pos={0 0}, upload=103]: Uploading piece... Jul 26 21:58:31.620258065 (6m47.401191437s) [upload=103]: Next returned nil segment Jul 26 21:58:31.620306924 (6m47.40124031s) [upload=103]: Waiting for error group managing segments... Jul 26 21:58:31.655784671 (6m47.436718042s) [noise=true , address=79.127.219.38:4005 , piece_id=NHI7KKCLXHQSSSJ4MEUEPZLVGFE7MW63FZPG7TEMEPKQ57E6TCXA, seg_pos={0 0}, upload=103]: Done uploading piece. err: <nil> Jul 26 21:58:32.448824802 (6m48.229758172s) [noise=true , address=79.127.201.215:4351 , piece_id=4YFZAWYDOBJWBC3ML5LRZCGZILV6RXGYZBQHPN7F3NHASJMSSS7Q, seg_pos={0 0}, upload=103]: Done uploading piece. err: <nil> Jul 26 21:58:37.263995765 (6m53.044929132s) [noise=true , address=77.90.31.16:4766 , piece_id=4N3DJG7Z7TGBLW4ELOB25SDN65UZUAIU5LEPILOFV3PQHSPYIUEA, seg_pos={0 0}, upload=103]: Done uploading piece. err: <nil> Jul 26 21:58:42.771973501 (6m58.552906875s) [noise=true , address=77.90.31.18:4833 , piece_id=57GAN7M3VDVGGTTSOO3VEVMEALFI7DOKCMBJ6CWTKDBIE2QH3DQA, seg_pos={0 0}, upload=103]: Done uploading piece. err: <nil> Jul 26 21:59:31.472005640 (7m47.252939024s) [noise=true , address=104.36.82.46:4494 , piece_id=S3B6RF3IHKB4GFGOTF6HJC2LNRNF3JMYIPUQBI5HGRRT33BGQLSA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: rpc: dial tcp 104.36.82.46:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472114083 (7m47.253047466s) [noise=true , address=104.36.82.10:4096 , piece_id=ZM65X7BOHPV556VUDHD6GP5KR37QS6RHEO3U3UZ7UREVBVKQTSJA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: rpc: dial tcp 104.36.82.10:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472356598 (7m47.25328998s) [noise=true , address=104.36.82.52:4118 , piece_id=BHLNV2EREN465VRGZGD7BNIHAVYHSQCDSBW7AEZQL2Y7OEGTJUHA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: rpc: dial tcp 104.36.82.52:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472586028 (7m47.253519411s) [noise=true , address=104.36.82.54:4191 , piece_id=CTZBMQF62W7ZTWYCC5LCB43NEIQIVIX7BXUOORM57Z5ZZAD2AZ7A, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 159.223.143.39:46830->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472670460 (7m47.253603836s) [noise=true , address=104.36.82.13:4212 , piece_id=MXYVFLXWHA32REPGP3PZ6XIHOJSLOT7UNQVSSYMXICUWRBNLNDLA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472743974 (7m47.253677353s) [noise=true , address=104.36.82.22:4529 , piece_id=A3O3SVOFVEWV3FKHMFK4C2LJQ3DBZVDQILQKWOZHJV6EU2MCPZFQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:12bNgTA4CY9ShB6Mi3QpceTEqNCr8ohCMHNVvU6S1qJp2TdVP1i, address:104.36.82.22:4529): protocol: write tcp 159.223.143.39:56532->104.36.82.22:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472807407 (7m47.253740783s) [noise=true , address=104.36.82.53:4124 , piece_id=YZBFYCO5UYONAGYAIOWSFTZVQPZT4Z2PVOOL3ARW7JMSWHZMTNNQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.472952319 (7m47.253885699s) [noise=true , address=104.36.82.47:4518 , piece_id=IIO4W6SNZNYDRAGQGZGK6UDC2JXCCIJNU7TRO4NPVCEYNXJJ2LZQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: write tcp 159.223.143.39:42860->104.36.82.47:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473038619 (7m47.253971998s) [noise=true , address=104.36.82.18:4397 , piece_id=AKS7NJGWMT2NCU4CGF5475JAX6TOX25XG33E6CAPPDJB4NGGOKXQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:12NkWHX2ojDAfX4CHoHSDE5mrLczUHwWQwm9rKPU7NUUB2GJgqY, address:104.36.82.18:4397): protocol: write tcp 159.223.143.39:52742->104.36.82.18:4397: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473102032 (7m47.254035414s) [noise=true , address=104.36.82.42:4800 , piece_id=DDJISQFLDSYZOI3TRNT2ZWDZUNAJR4GK4NS5YMHWLMV2MBYDQEDQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:121yHupysREmASSVx3WUN57duu16QEs7sA2BPBWyaThFzruvch2, address:104.36.82.42:4800): protocol: write tcp 159.223.143.39:53556->104.36.82.42:4800: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473195521 (7m47.254128887s) [noise=true , address=104.36.82.39:4735 , piece_id=FP2JFSR3OHXGETWGAMBTX5SSGJ2JFC74RYD2KD5IDQ3NJOYPOCLA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:1QCQZ8eFGLa5TAaAJJzUip8o5DAELzTBTm4UFbt3Am5yYWXAZF, address:104.36.82.39:4735): protocol: write tcp 159.223.143.39:41166->104.36.82.39:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473295358 (7m47.254228721s) [noise=true , address=104.36.82.55:4207 , piece_id=DFXMH5PH4N3Z2SQ62Z6VOPY7AQGLYSGHE3LC5SOFIOMFUSM5ZSDA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:33476->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473315654 (7m47.254249045s) [noise=true , address=104.36.82.3:4787 , piece_id=PBQK4YYE2ZTB3YJQOIPDC5H6UUXVYPTAFM6BWFKB6JAV36JECADA, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:104.36.82.3:4787): protocol: write tcp 159.223.143.39:45928->104.36.82.3:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473381214 (7m47.254314585s) [noise=true , address=104.36.82.49:4009 , piece_id=EFZCCYNDQKOJ4SO6TVRKCTFO6Z26CTKNRQ6QDLPOWYGHQD4J2NUQ, seg_pos={0 0}, upload=103]: Done uploading piece. err: ecclient: upload failed (node:12p2hucv6vZHowvgLEgMQLSzkE6ZCdvSQtp7Nzf3r6VpNbDmAAZ, address:104.36.82.49:4009): protocol: write tcp 159.223.143.39:43244->104.36.82.49:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:31.473422595 (7m47.254355981s) [seg_pos={0 0}, upload=103]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 21:59:31.504976247 (7m47.285909625s) [upload=103]: Done uploading segments... Jul 26 21:59:31.505002517 (7m47.285935892s) [upload=103]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.035244283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX..XXX] duration: 30.03128805s FAIL (2025-07-26 21:59:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== PdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802012332 (7m12.582945696s) [noise=true , address=104.36.82.13:4186 , piece_id=JK3XUENYVYRYGIXBI5GEKHO3CTB23XU4OPPOCSVOIL5Y3RYQCMQA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802031194 (7m12.582964575s) [seg_pos={0 0}, upload=100]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:58:56.828924440 (7m12.609857809s) [upload=100]: Done uploading segments... Jul 26 21:58:56.828953351 (7m12.609886727s) [upload=100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:58:56.928383445 (7m12.70931681s) [upload=112]: Starting upload Jul 26 21:58:56.928483411 (7m12.709416796s) [upload=112]: Done starting upload Jul 26 21:58:56.928599078 (7m12.709532445s) [upload=112]: Uploading segments... Jul 26 21:58:56.928846534 (7m12.7097799s) [upload=112]: Got next segment. Inline: false Jul 26 21:58:56.928903063 (7m12.709836429s) [upload=112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:58:56.928909218 (7m12.709842599s) [upload=112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:58:56.962397706 (7m12.743331082s) [seg_pos={0 0}, upload=112]: Begin upload segment... Jul 26 21:58:56.962446320 (7m12.743379685s) [seg_pos={0 0}, upload=112]: Done begin upload segment. Jul 26 21:58:56.962486844 (7m12.743420221s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962504965 (7m12.743438346s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962963964 (7m12.743897353s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.962989197 (7m12.743922579s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963409054 (7m12.744342436s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963423183 (7m12.744356568s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.963781349 (7m12.744714731s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964537876 (7m12.745471253s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.964930612 (7m12.745863994s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.965349574 (7m12.746282953s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.966552138 (7m12.74748552s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967124362 (7m12.74805774s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.967724725 (7m12.748658107s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968251177 (7m12.749184569s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.968796893 (7m12.749730277s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:56.969667421 (7m12.75060079s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Uploading piece... Jul 26 21:58:57.057172667 (7m12.838106037s) [upload=112]: Next returned nil segment Jul 26 21:58:57.057190660 (7m12.838124044s) [upload=112]: Waiting for error group managing segments... Jul 26 21:58:57.061079888 (7m12.842013253s) [noise=true , address=45.91.116.4:4937 , piece_id=FYEDDFUKEBWEINWHQ5TBO2MPDBAVRK5EMU4RJVFJJL5DVNP4WERQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.061664155 (7m12.842597519s) [noise=true , address=45.91.116.10:4082 , piece_id=RQCR2JGHGTDTIFPDH7EOYKRGENZPQJSJXD7VQAE3I3BYRBH2HXIA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062174746 (7m12.843108108s) [noise=true , address=45.91.116.12:4226 , piece_id=G47XBQO3INPZYHMZSV2UAMY75LYBEP4YAQITXMGFGDJCBEV2GP7A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.062588806 (7m12.843522176s) [noise=true , address=45.91.116.4:4938 , piece_id=52UWMPRD2XNIIWQ43TOBHB7DQ22DY2K4T56LIWZQAVXFRP7NZN6Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063003156 (7m12.84393652s) [noise=true , address=45.91.116.6:4975 , piece_id=4DSOCLFMCZD74PLLRMFQKPAIBN2GGXI7LYQAJ6IPSQXJA54ILZWQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063537160 (7m12.844470526s) [noise=true , address=45.91.116.5:4297 , piece_id=OTOJRGILL44W7BPBDR4MPGB37RSRUEF4TMWTJ2EUCPQ7SCFQM5GQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063593642 (7m12.844527007s) [noise=true , address=45.91.116.11:4118 , piece_id=M24QJTS5V6GGGBOKBWQPLOCKJRG5ZQDBNVU7Z7EWRD2WQSNFVDEQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.063923281 (7m12.844856646s) [noise=true , address=45.91.116.9:4262 , piece_id=IZF4KEHD5ZO4EQUE6FDZ6I6Z6VYQ3NVKRZCMTAYQ5P3R26NLCXPA, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:58:57.064066458 (7m12.844999823s) [noise=true , address=45.91.116.7:4009 , piece_id=TOZJEGVQ3B775GIDQLF6OIHUGSZZHC6ZJVY647KO43ELN4P5P6RQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:00.371963705 (7m16.15289707s) [noise=true , address=77.90.31.11:4555 , piece_id=DBUZ4JPBKNLSDZQX767QLUKO34WLHIFMZAQEOQEXNWBV4Z2M5P6A, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:05.374869080 (7m21.155802458s) [noise=true , address=77.90.31.13:4626 , piece_id=KYO2YOSQTGP7MI6NQN7LOQLB4VJSZRCUYW4EGSHINCC2XC77QD4Q, seg_pos={0 0}, upload=112]: Done uploading piece. err: <nil> Jul 26 21:59:26.900101378 (7m42.681034761s) [noise=true , address=104.36.82.7:4911 , piece_id=ASL6PY7BNZMBWXNUG553ULH4HZNEV2TDGAKYDYJYUR576HGFKLFA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:54702->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900206538 (7m42.681139914s) [noise=true , address=104.36.82.14:4221 , piece_id=VMVXYLDQHGWMWIBQBQS4THGL4VAETYBABLU55P3SY5BLE2UESJLA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:56252->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900260676 (7m42.681194052s) [noise=true , address=104.36.82.8:4971 , piece_id=EWMN3YUOUSAC5CX5DESBIO56B625P6ORRAN75Z7SUSRVU5TL75SA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:43016->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900359586 (7m42.681292969s) [noise=true , address=104.36.82.3:4765 , piece_id=KWSVEW7NHQCGQHEDL5M4LN6XBO6GZ6FJMCG6ONXBFZCYAX4646CQ, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900429335 (7m42.681362712s) [noise=true , address=104.36.82.9:4005 , piece_id=ZN5IBVKNJ4ZIJGOBHVZCCSLIECF7KPPRMGCEFSSPBSR5ANCHYZAA, seg_pos={0 0}, upload=112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:55284->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:26.900454028 (7m42.681387411s) [seg_pos={0 0}, upload=112]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:59:26.927954817 (7m42.708888207s) [upload=112]: Done uploading segments... Jul 26 21:59:26.927985802 (7m42.708919184s) [upload=112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031342248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X..X....] duration: 4.452623747s FAIL (2025-07-26 21:59:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.452661099s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX..XX] duration: 30.031727824s FAIL (2025-07-26 21:58:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== emTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:54482->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:56.338859183 (6m12.11979256s) [noise=true , address=104.36.82.7:4909 , piece_id=MGO2JEQV3M5RNSTYEX2GHER5KVCT57LYROGUJ77GKTICNMGLTC5A, seg_pos={0 0}, upload=86]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:37530->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:56.338900056 (6m12.119833436s) [seg_pos={0 0}, upload=86]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:57:56.368868105 (6m12.14980149s) [upload=86]: Done uploading segments... Jul 26 21:57:56.368901894 (6m12.149835276s) [upload=86]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 21:58:26.831070143 (6m42.612003527s) [upload=100]: Starting upload Jul 26 21:58:26.831153943 (6m42.61208733s) [upload=100]: Done starting upload Jul 26 21:58:26.831233128 (6m42.612166497s) [upload=100]: Uploading segments... Jul 26 21:58:26.831515202 (6m42.612448587s) [upload=100]: Got next segment. Inline: false Jul 26 21:58:26.831577924 (6m42.612511306s) [upload=100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:58:26.831584954 (6m42.612518337s) [upload=100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:58:26.860812943 (6m42.641746329s) [seg_pos={0 0}, upload=100]: Begin upload segment... Jul 26 21:58:26.860867713 (6m42.641801099s) [seg_pos={0 0}, upload=100]: Done begin upload segment. Jul 26 21:58:26.860897617 (6m42.641831s) [noise=true , address=104.36.82.14:4221 , piece_id=HOTTQLB2KSPD2KNR3H2LYH4CG37BLTYNOIHBYTSTZA4K7MNZ73TA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.861377438 (6m42.642310823s) [noise=true , address=45.91.116.4:4937 , piece_id=T4QRUK6TVV3F6QM32B3RP3J2GQZ655OOOIYJXKUX3YQMFWCFL3CQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.861756562 (6m42.642689947s) [noise=true , address=45.91.116.6:4975 , piece_id=HZU2NITYSRK2ZTWCKJE33DYXOT6ZFGER4UDTG5CRRBGP5W7567AQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.862126401 (6m42.643059789s) [noise=true , address=45.91.116.7:4010 , piece_id=AG3IUGBK77NQLCP435QMVSXM7OHU43CQ5ZC7QORD5VR52SAVZ5RQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.862462692 (6m42.643396078s) [noise=true , address=104.36.82.16:4293 , piece_id=2OIXLDI2KMOVKMK5DUUQAJX3YSTF76UNRVTYVH4GS3WSGRJSNQGQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.862830282 (6m42.643763667s) [noise=true , address=104.36.82.10:4078 , piece_id=RMUO4EPV6MCHV26N2BWR3PU6XIFZISLUGR4TAPIJLB3VKIZGUAEA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.863359518 (6m42.644292904s) [noise=true , address=45.91.116.13:4154 , piece_id=QT26AKZMARETSPARLXIVXBJYXZR73NGGVSTXRS7D22KYX6STNWWQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.863845330 (6m42.644778711s) [noise=true , address=45.91.116.12:4227 , piece_id=JGTNMRJQLHQHXMZKFW77MVOKH24L67KDPNEF4E7DBNEZCJ73ZYJQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.864275837 (6m42.645209212s) [noise=true , address=45.91.116.9:4263 , piece_id=S5ALGZLMWTCOK4HR6F2BAF4D2SGFOENSIZNF6RSAXKVIJ2FNK7GA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.866797914 (6m42.647731301s) [noise=true , address=104.36.82.13:4186 , piece_id=JK3XUENYVYRYGIXBI5GEKHO3CTB23XU4OPPOCSVOIL5Y3RYQCMQA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.868248679 (6m42.649182062s) [noise=true , address=45.91.116.4:4938 , piece_id=2MPLM6IVN6TTAHN5ILFOZYC6OYAPAT52ILKBYLGYDNL75F3WVO3Q, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.868628777 (6m42.649562138s) [noise=true , address=45.91.116.9:4261 , piece_id=GHT2VV7SAWYARTOLNHPNLT2QGEXTQT5XHYVF2XT66VDZA22MM6VQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.868923707 (6m42.649857091s) [noise=true , address=45.91.116.4:4939 , piece_id=W75DUIW7L332KAYC3UW3INFNJW5GWBVBUHG5SOEWCUCZCYZALVSQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.869352668 (6m42.650286046s) [noise=true , address=45.91.116.5:4297 , piece_id=MDZNSYI6W7DXVVFHVWLXT4Q74FBKRK5ZLPRJVM5ZGIMII7IHOMSA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.869856486 (6m42.65078987s) [noise=true , address=45.91.116.10:4081 , piece_id=GKGURF7GZLJXEHZFJMRY32QWTGPQV56O5BLW4DJWSZ2L46VH5IMA, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.870315120 (6m42.651248507s) [noise=true , address=104.36.82.7:4910 , piece_id=BL3SELE76Y4GX454H45RP2QZJDFEZVEPL5VOLWEANHSFNCBQHKUQ, seg_pos={0 0}, upload=100]: Uploading piece... Jul 26 21:58:26.958466486 (6m42.739399863s) [upload=100]: Next returned nil segment Jul 26 21:58:26.958482402 (6m42.739415782s) [upload=100]: Waiting for error group managing segments... Jul 26 21:58:26.961157566 (6m42.74209095s) [noise=true , address=45.91.116.10:4081 , piece_id=GKGURF7GZLJXEHZFJMRY32QWTGPQV56O5BLW4DJWSZ2L46VH5IMA, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.961716915 (6m42.742650295s) [noise=true , address=45.91.116.4:4939 , piece_id=W75DUIW7L332KAYC3UW3INFNJW5GWBVBUHG5SOEWCUCZCYZALVSQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.961783562 (6m42.742716944s) [noise=true , address=45.91.116.6:4975 , piece_id=HZU2NITYSRK2ZTWCKJE33DYXOT6ZFGER4UDTG5CRRBGP5W7567AQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.962301395 (6m42.743234764s) [noise=true , address=45.91.116.5:4297 , piece_id=MDZNSYI6W7DXVVFHVWLXT4Q74FBKRK5ZLPRJVM5ZGIMII7IHOMSA, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.962328267 (6m42.743261651s) [noise=true , address=45.91.116.4:4937 , piece_id=T4QRUK6TVV3F6QM32B3RP3J2GQZ655OOOIYJXKUX3YQMFWCFL3CQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.962877695 (6m42.743811065s) [noise=true , address=45.91.116.9:4261 , piece_id=GHT2VV7SAWYARTOLNHPNLT2QGEXTQT5XHYVF2XT66VDZA22MM6VQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.962984943 (6m42.743918331s) [noise=true , address=45.91.116.12:4227 , piece_id=JGTNMRJQLHQHXMZKFW77MVOKH24L67KDPNEF4E7DBNEZCJ73ZYJQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.963539676 (6m42.744473044s) [noise=true , address=45.91.116.4:4938 , piece_id=2MPLM6IVN6TTAHN5ILFOZYC6OYAPAT52ILKBYLGYDNL75F3WVO3Q, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.963551083 (6m42.744484478s) [noise=true , address=45.91.116.7:4010 , piece_id=AG3IUGBK77NQLCP435QMVSXM7OHU43CQ5ZC7QORD5VR52SAVZ5RQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.964024675 (6m42.744958042s) [noise=true , address=45.91.116.9:4263 , piece_id=S5ALGZLMWTCOK4HR6F2BAF4D2SGFOENSIZNF6RSAXKVIJ2FNK7GA, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:26.995375683 (6m42.776309061s) [noise=true , address=45.91.116.13:4154 , piece_id=QT26AKZMARETSPARLXIVXBJYXZR73NGGVSTXRS7D22KYX6STNWWQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: <nil> Jul 26 21:58:56.801647750 (7m12.582581125s) [noise=true , address=104.36.82.14:4221 , piece_id=HOTTQLB2KSPD2KNR3H2LYH4CG37BLTYNOIHBYTSTZA4K7MNZ73TA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:56.801714425 (7m12.58264779s) [noise=true , address=104.36.82.16:4293 , piece_id=2OIXLDI2KMOVKMK5DUUQAJX3YSTF76UNRVTYVH4GS3WSGRJSNQGQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:38518->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:56.801848128 (7m12.582781504s) [noise=true , address=104.36.82.7:4910 , piece_id=BL3SELE76Y4GX454H45RP2QZJDFEZVEPL5VOLWEANHSFNCBQHKUQ, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:53416->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:56.801924275 (7m12.582857649s) [noise=true , address=104.36.82.10:4078 , piece_id=RMUO4EPV6MCHV26N2BWR3PU6XIFZISLUGR4TAPIJLB3VKIZGUAEA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802012332 (7m12.582945696s) [noise=true , address=104.36.82.13:4186 , piece_id=JK3XUENYVYRYGIXBI5GEKHO3CTB23XU4OPPOCSVOIL5Y3RYQCMQA, seg_pos={0 0}, upload=100]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:58:56.802031194 (7m12.582964575s) [seg_pos={0 0}, upload=100]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:58:56.828924440 (7m12.609857809s) [upload=100]: Done uploading segments... Jul 26 21:58:56.828953351 (7m12.609886727s) [upload=100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031755126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration: 2.362994608s FAIL (2025-07-26 21:58:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.363023393s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.043588306s FAIL (2025-07-26 21:58:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== eg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.422219637 (5m47.203153012s) [noise=true , address=104.36.82.21:4508 , piece_id=CY6BMV4BXSB4C6K2ZHQQPMLTBNBSGSWFQDUCU3LXGVGSYTTEITGA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.423692607 (5m47.204625977s) [seg_pos={0 0}, upload=87]: Done begin upload segment. Jul 26 21:57:31.423710251 (5m47.204643613s) [noise=true , address=79.127.205.234:4614 , piece_id=53OW2XWYRSJNJW5L7E5AMNROGFUH3IMSSATPHN37I4CGKHRB23SA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.425336303 (5m47.206269686s) [noise=true , address=104.36.82.11:4144 , piece_id=UVAD3U3G6JESCRJGNPUTVYXNW5KSM5JAXMXOFRMSJMLNAAURYVFA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.425983915 (5m47.206917278s) [noise=true , address=104.36.82.54:4180 , piece_id=JJMRUN53AMXY5JX2BJOOUCKDTGAQRKCZEZLVXUYDYCUATAQRJHAQ, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.426616891 (5m47.207550253s) [noise=true , address=104.36.82.9:4015 , piece_id=C2YAEI7IZIUXPGERWNBXTC6ZIM7RPFFQ24D7S6MYD45YUQKJIOZQ, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.431296685 (5m47.21223007s) [noise=true , address=79.127.205.235:4659 , piece_id=FN6OURUEMDW5ICXHVMJT4IXUKJJFBYRWMSZ6HYM4XBBBVKVGL6KA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.431763678 (5m47.212697065s) [noise=true , address=64.49.13.138:4483 , piece_id=MGEQHY4QZDXHGCBQET6M55GWBN6EA7EF5RANIJUB3JFIKN7EFBMQ, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.432204500 (5m47.21313789s) [noise=true , address=104.36.82.47:4523 , piece_id=B7KR5UOIQZVND2NNCM6ZJR3GWDTWY62ENF7JJFDLW2AHGWJJFVVA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.432586902 (5m47.213520287s) [noise=true , address=104.36.82.37:4672 , piece_id=V5SYHU3TENGY2VGQ2BQG25NTYJNDUI76XNYBNXULMKER6PIITSPA, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.432976862 (5m47.213910248s) [noise=true , address=79.127.219.38:4017 , piece_id=NVJJBVZ767UEZ2TSUY5C4WFLPAIENBDU2KEGCNPSZC2H5EVNRC6Q, seg_pos={0 0}, upload=87]: Uploading piece... Jul 26 21:57:31.495211905 (5m47.276145289s) [upload=87]: Next returned nil segment Jul 26 21:57:31.495233051 (5m47.276166437s) [upload=87]: Waiting for error group managing segments... Jul 26 21:57:31.598834306 (5m47.379767675s) [noise=true , address=79.127.219.38:4017 , piece_id=NVJJBVZ767UEZ2TSUY5C4WFLPAIENBDU2KEGCNPSZC2H5EVNRC6Q, seg_pos={0 0}, upload=87]: Done uploading piece. err: <nil> Jul 26 21:57:32.016808440 (5m47.797741816s) [noise=true , address=79.127.205.235:4659 , piece_id=FN6OURUEMDW5ICXHVMJT4IXUKJJFBYRWMSZ6HYM4XBBBVKVGL6KA, seg_pos={0 0}, upload=87]: Done uploading piece. err: <nil> Jul 26 21:57:32.240453403 (5m48.021386781s) [noise=true , address=79.127.205.234:4614 , piece_id=53OW2XWYRSJNJW5L7E5AMNROGFUH3IMSSATPHN37I4CGKHRB23SA, seg_pos={0 0}, upload=87]: Done uploading piece. err: <nil> Jul 26 21:57:33.276434882 (5m49.057368272s) [noise=true , address=64.49.13.138:4483 , piece_id=MGEQHY4QZDXHGCBQET6M55GWBN6EA7EF5RANIJUB3JFIKN7EFBMQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 21:58:31.351932871 (6m47.132866248s) [noise=true , address=104.36.82.46:4507 , piece_id=6FD3PYZLWX4OSU633Z7AIMIEBLD3H7OZ5GPK7CIAEID26G6IKKZQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352017706 (6m47.132951084s) [noise=true , address=104.36.82.21:4508 , piece_id=CY6BMV4BXSB4C6K2ZHQQPMLTBNBSGSWFQDUCU3LXGVGSYTTEITGA, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: rpc: dial tcp 104.36.82.21:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352309336 (6m47.133242721s) [noise=true , address=104.36.82.38:4713 , piece_id=KJW7WFQQCNK6UF4SFZV2S6FYUWC7Q47R6RVUF7E4AZUHQE6CTQXQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF, address:104.36.82.38:4713): protocol: rpc: dial tcp 104.36.82.38:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352396835 (6m47.133330211s) [noise=true , address=104.36.82.21:4493 , piece_id=FP5OZZEFE7ITF3I5WYNG7KJ22BRRJHTFRFA2GUCR3TQOEXLLVGUQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: write tcp 159.223.143.39:48552->104.36.82.21:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352449035 (6m47.133382416s) [noise=true , address=104.36.82.54:4180 , piece_id=JJMRUN53AMXY5JX2BJOOUCKDTGAQRKCZEZLVXUYDYCUATAQRJHAQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352688117 (6m47.133621497s) [noise=true , address=104.36.82.56:4259 , piece_id=COZ2ITRBE6ZJLAKAOMVHMN7MBQCSIQ5LY2XL7YNPN6YZYQQW2ETQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: rpc: dial tcp 104.36.82.56:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352751164 (6m47.133684543s) [noise=true , address=104.36.82.16:4307 , piece_id=72OA3QYBPN2WHYGXDWHSUSIUHCSRYFXQLGVTHNGAP3ATFZOAJRJA, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12QGfzeTEErAaE5YRTLi1w7SZoJbo2xVKb1Sykj1pW5fbCZYSCn, address:104.36.82.16:4307): protocol: rpc: dial tcp 104.36.82.16:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.352805136 (6m47.13373852s) [noise=true , address=104.36.82.11:4144 , piece_id=UVAD3U3G6JESCRJGNPUTVYXNW5KSM5JAXMXOFRMSJMLNAAURYVFA, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:128246NkJjT3UFiQdSqFssY6jTAV1jye3dXAjAkFnC3DUAXb235, address:104.36.82.11:4144): protocol: rpc: dial tcp 104.36.82.11:4144: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.353659952 (6m47.134593328s) [noise=true , address=104.36.82.47:4523 , piece_id=B7KR5UOIQZVND2NNCM6ZJR3GWDTWY62ENF7JJFDLW2AHGWJJFVVA, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 159.223.143.39:45248->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.353732025 (6m47.134665407s) [noise=true , address=104.36.82.37:4672 , piece_id=V5SYHU3TENGY2VGQ2BQG25NTYJNDUI76XNYBNXULMKER6PIITSPA, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1G9dUkPkLRrEkMGsTiaN9iB8YnZ9GF7moCUPhGiu1XDDi8XnSu, address:104.36.82.37:4672): protocol: write tcp 159.223.143.39:59392->104.36.82.37:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.353792541 (6m47.134725917s) [noise=true , address=104.36.82.5:4871 , piece_id=PWYCHKAHQQNKR4ITWWCW3SAXY7QMUVL7ZT4Q2BIKW5TO5GQ7HV2Q, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1NEVGFZtdgMqREnYBPuBbYqwGRxExenRwncwCBLDFhuwdFyWdb, address:104.36.82.5:4871): protocol: write tcp 159.223.143.39:57648->104.36.82.5:4871: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.353851715 (6m47.134785091s) [noise=true , address=104.36.82.53:4155 , piece_id=QGKVQ5TKXHUHDGYBHVXUSTB6P7EPXVLFCJCTUOQ6KL3GT5LRJT7A, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.354058493 (6m47.134991881s) [noise=true , address=104.36.82.9:4022 , piece_id=TPTQEYAFECORD6MDJ24L6ZT34TFUIC34VBZPMP24BEUT5SN5U3AQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: write tcp 159.223.143.39:39642->104.36.82.9:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.354100526 (6m47.135033901s) [noise=true , address=104.36.82.9:4015 , piece_id=C2YAEI7IZIUXPGERWNBXTC6ZIM7RPFFQ24D7S6MYD45YUQKJIOZQ, seg_pos={0 0}, upload=87]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: write tcp 159.223.143.39:39646->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:31.354148100 (6m47.135081481s) [seg_pos={0 0}, upload=87]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 26 21:58:31.392531954 (6m47.173465333s) [upload=87]: Done uploading segments... Jul 26 21:58:31.392563658 (6m47.173497039s) [upload=87]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.043606617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X..X..] duration: 8.923610961s FAIL (2025-07-26 21:58:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.923651487s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration: 30.0328715s FAIL (2025-07-26 21:57:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== t canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: rpc: dial tcp 104.36.82.5:4839: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:01.657254325 (5m17.438187693s) [noise=true , address=104.36.82.16:4295 , piece_id=EBYOIFUHC5YJED2ATKTVT6OJUV3LNHBVFDY57IGF7UG4G6EGFTGA, seg_pos={0 0}, upload=78]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 159.223.143.39:44464->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:01.657271506 (5m17.438204887s) [seg_pos={0 0}, upload=78]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:57:01.657373497 (5m17.438306884s) [upload=78]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:57:01.657381451 (5m17.438314838s) [upload=78]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:57:01.708939449 (5m17.489872827s) [upload=78]: Done uploading segments... Jul 26 21:57:01.708963735 (5m17.489897115s) [upload=78]: Upload finished. err: <nil> Jul 26 21:57:26.371430273 (5m42.152363645s) [upload=86]: Starting upload Jul 26 21:57:26.371535873 (5m42.152469252s) [upload=86]: Done starting upload Jul 26 21:57:26.371615149 (5m42.152548529s) [upload=86]: Uploading segments... Jul 26 21:57:26.371955139 (5m42.152888504s) [upload=86]: Got next segment. Inline: false Jul 26 21:57:26.371989073 (5m42.152922448s) [upload=86]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:57:26.371994527 (5m42.152927902s) [upload=86]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:57:26.403564732 (5m42.184498119s) [seg_pos={0 0}, upload=86]: Begin upload segment... Jul 26 21:57:26.403642238 (5m42.184575618s) [seg_pos={0 0}, upload=86]: Done begin upload segment. Jul 26 21:57:26.403675944 (5m42.18460933s) [noise=true , address=45.91.116.4:4937 , piece_id=YZ6YDAI4XGISP44AMPXJWMXCWBIOPPMNFFBVLLEA7AJKRBHEGEIA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.403738689 (5m42.184672077s) [noise=true , address=45.91.116.7:4011 , piece_id=S3AI3QMLRSSGIWVMQMIXO4QAMXFG2DI3LFIWB5Z2ICK6VNIMY2ZA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.404393487 (5m42.185326911s) [noise=true , address=45.91.116.10:4082 , piece_id=CUOZM442LOOM2EZ7T35Q5S3ZFC22NLU7RYNMPHRPJNF6FK6OYGEQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.404489510 (5m42.185422898s) [noise=true , address=45.91.116.13:4154 , piece_id=Z47AEYOF3GRQ66HP737PJ4R6LLL4VQESCAYQPXDWWCSQYQ3S6D2Q, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.404796004 (5m42.185729391s) [noise=true , address=45.91.116.7:4010 , piece_id=SPVON4WCUTSBBCX3J5ZELAWV2JHG6ENRMQGHLX3OKNACRF2KD4OA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.405005346 (5m42.185938733s) [noise=true , address=45.91.116.8:4048 , piece_id=OFAWGG6LOX4C34WU7C2H3WKYLG3UGNQI4CE6XJ6DHYUJF7Q34XPA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.405153128 (5m42.186086512s) [noise=true , address=104.36.82.6:4876 , piece_id=NXQ3ISXAWRUKJAZFJQV64LU2A6IRK2CS5GK5BKDKC75AOTI2SOTA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.405523052 (5m42.186456441s) [noise=true , address=45.91.116.12:4227 , piece_id=Y6FCSWPBRWVXB4OG6TQ4MDRCODRXIER27YQUZ4O2LMUCJCFR7PZQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.405534598 (5m42.186467983s) [noise=true , address=104.36.82.15:4257 , piece_id=43LWA6V4CZJQW5QMYNDXNGVWDTPFZBEGCD6C7GLO7WBS6SMEQO4A, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.405871948 (5m42.186805329s) [noise=true , address=77.90.31.10:4518 , piece_id=VMLI25YIPSQJSBPNEPSJPFMBUJEKO6HLA37USZXENWQYDDYJG5ZQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.406017937 (5m42.186951325s) [noise=true , address=45.91.116.6:4974 , piece_id=N5TDL6RUJIY2HJZBCWKAVKYD2RGEAYE2TVXZQI4GJM4EREOWWRFQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.406182145 (5m42.187115523s) [noise=true , address=104.36.82.8:4971 , piece_id=H4H5VX2OCSQ5OKW724ZAFUTFUMKO3VGQSCWUMWDMWUXJMOGYXRMQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.406501960 (5m42.187435346s) [noise=true , address=104.36.82.7:4909 , piece_id=MGO2JEQV3M5RNSTYEX2GHER5KVCT57LYROGUJ77GKTICNMGLTC5A, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.411538395 (5m42.192471791s) [noise=true , address=45.91.116.13:4153 , piece_id=PULBDNEGHWR34NOIY3KV6GVKCETEXO5PMHY6OJFIXE5UNCSJFUCQ, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.412063785 (5m42.192997173s) [noise=true , address=45.91.116.9:4263 , piece_id=KTEANMJAABWCNMQ464MD5LCQQIUQIRYUIFSSYJFKJUWHFM6RY6CA, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.413829345 (5m42.194762734s) [noise=true , address=45.91.116.5:4298 , piece_id=PB4WCOPLGU35VXFLINKK223FQWC7MEXDSZJNOXP2RAIOQZBFPZ4Q, seg_pos={0 0}, upload=86]: Uploading piece... Jul 26 21:57:26.521735333 (5m42.302668729s) [upload=86]: Next returned nil segment Jul 26 21:57:26.521758809 (5m42.302692196s) [upload=86]: Waiting for error group managing segments... Jul 26 21:57:26.525707631 (5m42.306641001s) [noise=true , address=45.91.116.12:4227 , piece_id=Y6FCSWPBRWVXB4OG6TQ4MDRCODRXIER27YQUZ4O2LMUCJCFR7PZQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.525753862 (5m42.306687225s) [noise=true , address=45.91.116.8:4048 , piece_id=OFAWGG6LOX4C34WU7C2H3WKYLG3UGNQI4CE6XJ6DHYUJF7Q34XPA, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.526803203 (5m42.307736579s) [noise=true , address=45.91.116.9:4263 , piece_id=KTEANMJAABWCNMQ464MD5LCQQIUQIRYUIFSSYJFKJUWHFM6RY6CA, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.526857134 (5m42.30779051s) [noise=true , address=45.91.116.4:4937 , piece_id=YZ6YDAI4XGISP44AMPXJWMXCWBIOPPMNFFBVLLEA7AJKRBHEGEIA, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.527348493 (5m42.30828186s) [noise=true , address=45.91.116.13:4154 , piece_id=Z47AEYOF3GRQ66HP737PJ4R6LLL4VQESCAYQPXDWWCSQYQ3S6D2Q, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.527435900 (5m42.308369275s) [noise=true , address=45.91.116.7:4010 , piece_id=SPVON4WCUTSBBCX3J5ZELAWV2JHG6ENRMQGHLX3OKNACRF2KD4OA, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.527859034 (5m42.308792413s) [noise=true , address=45.91.116.6:4974 , piece_id=N5TDL6RUJIY2HJZBCWKAVKYD2RGEAYE2TVXZQI4GJM4EREOWWRFQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.527924698 (5m42.308858065s) [noise=true , address=45.91.116.13:4153 , piece_id=PULBDNEGHWR34NOIY3KV6GVKCETEXO5PMHY6OJFIXE5UNCSJFUCQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.528526364 (5m42.309459727s) [noise=true , address=45.91.116.5:4298 , piece_id=PB4WCOPLGU35VXFLINKK223FQWC7MEXDSZJNOXP2RAIOQZBFPZ4Q, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.528643859 (5m42.309577243s) [noise=true , address=45.91.116.10:4082 , piece_id=CUOZM442LOOM2EZ7T35Q5S3ZFC22NLU7RYNMPHRPJNF6FK6OYGEQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:26.529062271 (5m42.309995638s) [noise=true , address=45.91.116.7:4011 , piece_id=S3AI3QMLRSSGIWVMQMIXO4QAMXFG2DI3LFIWB5Z2ICK6VNIMY2ZA, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:31.196523090 (5m46.977456467s) [noise=true , address=77.90.31.10:4518 , piece_id=VMLI25YIPSQJSBPNEPSJPFMBUJEKO6HLA37USZXENWQYDDYJG5ZQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: <nil> Jul 26 21:57:56.338588603 (6m12.119521983s) [noise=true , address=104.36.82.15:4257 , piece_id=43LWA6V4CZJQW5QMYNDXNGVWDTPFZBEGCD6C7GLO7WBS6SMEQO4A, seg_pos={0 0}, upload=86]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:56.338662558 (6m12.119595935s) [noise=true , address=104.36.82.8:4971 , piece_id=H4H5VX2OCSQ5OKW724ZAFUTFUMKO3VGQSCWUMWDMWUXJMOGYXRMQ, seg_pos={0 0}, upload=86]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:57:56.338737387 (6m12.119670761s) [noise=true , address=104.36.82.6:4876 , piece_id=NXQ3ISXAWRUKJAZFJQV64LU2A6IRK2CS5GK5BKDKC75AOTI2SOTA, seg_pos={0 0}, upload=86]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:54482->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:56.338859183 (6m12.11979256s) [noise=true , address=104.36.82.7:4909 , piece_id=MGO2JEQV3M5RNSTYEX2GHER5KVCT57LYROGUJ77GKTICNMGLTC5A, seg_pos={0 0}, upload=86]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:37530->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:56.338900056 (6m12.119833436s) [seg_pos={0 0}, upload=86]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:57:56.368868105 (6m12.14980149s) [upload=86]: Done uploading segments... Jul 26 21:57:56.368901894 (6m12.149835276s) [upload=86]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.032902332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.032945173s FAIL (2025-07-26 21:57:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 9859962 (4m47.11079334s) [noise=true , address=104.36.82.47:4542 , piece_id=IPQK4AZXRERR2LC4F4EYGJKRFFTNO6EY4XU3W5DDFVU3ZQAQCHMA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.330207000 (4m47.111140379s) [noise=true , address=104.36.82.12:4170 , piece_id=VVBD45K2DZGLE5XHTVUJVT6QY6NUDHCRSNPNFPDMWDNC7HYRXOLA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.330543115 (4m47.111476501s) [noise=true , address=104.36.82.46:4505 , piece_id=CMS2W2HEXZNUOQRTYRXWEILOF7CCHM74MYUCSYUVTS4C76BTFMYA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.330942428 (4m47.111875813s) [noise=true , address=104.36.82.13:4191 , piece_id=55E5CZ54YW5CDKXV76VAAJB5LH3OYJWFTV5OKIX7EMCX72MLHNOA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.331431957 (4m47.112365338s) [noise=true , address=45.91.116.8:4060 , piece_id=LCJNZSHWM3D6V4BSIZEN4BW4MDX6NGNGVHH5FHLRZXOR6LIRMUHA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.331788479 (4m47.112721861s) [noise=true , address=79.127.205.246:4285 , piece_id=SJO5AFKZBWVGJ2RVJP3C24FHGSH6HAMMIS4YLIT577CCNMSKKIKQ, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.332174558 (4m47.113107952s) [noise=true , address=79.127.201.215:4353 , piece_id=2XB467JOLGOJX47VQON2CAC6T5XAY7XV3ESBBYIUYT7IPCWBQD3A, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.332578501 (4m47.113511885s) [noise=true , address=104.36.82.11:4118 , piece_id=GVKJKH5GFRGVFOV7WKBQXQLJLDSDBKJG4GLYHEQ6YBDI45AH6VVQ, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.332941042 (4m47.113874423s) [noise=true , address=104.36.82.55:4228 , piece_id=CJRXNCJOXEMFUHNDOFZEEP6UME67ZUSUHTWR6JDXU3DW22FBR63Q, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.333281921 (4m47.114215299s) [noise=true , address=104.36.82.56:4230 , piece_id=WKUQHC3I6S2D564EYCGSMN5QTGT3B2EMTGVQAROQ2NOEQAIBIF5Q, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.333591743 (4m47.114525121s) [noise=true , address=104.36.82.52:4096 , piece_id=H2KK43MRWWADOPPWG655WQN45CYYP4W6SJVFBWDPLBDGJOG4C6RQ, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.333888083 (4m47.114821463s) [noise=true , address=79.127.205.245:4246 , piece_id=XK4QMBOX3ARCPSNOKTUSCY6WNB3WS7RYEC6B3JSUEF5J4FHYZAWA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.334241163 (4m47.115174548s) [noise=true , address=77.90.31.13:4634 , piece_id=INXSK3KXE2M4KCQKU5ZK5QYHA63672XNFR4KXL54ILYCQEIYXCDA, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.334645591 (4m47.115578974s) [noise=true , address=104.36.82.11:4133 , piece_id=NXAZFSICGF3N3KRVNFI7O6BLPLXV4HXXLRZB45DO4EIYPCZ7MWSQ, seg_pos={0 0}, upload=74]: Uploading piece... Jul 26 21:56:31.547992231 (4m47.32892561s) [upload=74]: Next returned nil segment Jul 26 21:56:31.548638545 (4m47.329571924s) [upload=74]: Waiting for error group managing segments... Jul 26 21:56:31.993483248 (4m47.774416627s) [noise=true , address=79.127.205.245:4246 , piece_id=XK4QMBOX3ARCPSNOKTUSCY6WNB3WS7RYEC6B3JSUEF5J4FHYZAWA, seg_pos={0 0}, upload=74]: Done uploading piece. err: <nil> Jul 26 21:56:32.052961808 (4m47.833895171s) [noise=true , address=79.127.205.246:4285 , piece_id=SJO5AFKZBWVGJ2RVJP3C24FHGSH6HAMMIS4YLIT577CCNMSKKIKQ, seg_pos={0 0}, upload=74]: Done uploading piece. err: <nil> Jul 26 21:56:34.440805540 (4m50.221738924s) [noise=true , address=45.91.116.8:4060 , piece_id=LCJNZSHWM3D6V4BSIZEN4BW4MDX6NGNGVHH5FHLRZXOR6LIRMUHA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1WiDhEQp7L11dVHj9okUMqG5Ty6Sy3uhVrEkipspBX39Pu9mRX, address:45.91.116.8:4060): protocol: storage node overloaded, request limit: 2000 Jul 26 21:56:36.808729318 (4m52.589662698s) [noise=true , address=77.90.31.13:4634 , piece_id=INXSK3KXE2M4KCQKU5ZK5QYHA63672XNFR4KXL54ILYCQEIYXCDA, seg_pos={0 0}, upload=74]: Done uploading piece. err: <nil> Jul 26 21:56:36.916365631 (4m52.697299021s) [noise=true , address=64.49.13.170:4721 , piece_id=FVDLU3OABYIHQ72HI3OKZCXF3N6OMBTOEKQZCUXYPRA2D4PB3Q7Q, seg_pos={0 0}, upload=74]: Done uploading piece. err: <nil> Jul 26 21:56:38.095294762 (4m53.876228128s) [noise=true , address=79.127.201.215:4353 , piece_id=2XB467JOLGOJX47VQON2CAC6T5XAY7XV3ESBBYIUYT7IPCWBQD3A, seg_pos={0 0}, upload=74]: Done uploading piece. err: <nil> Jul 26 21:57:31.242960556 (5m47.023893918s) [noise=true , address=104.36.82.54:4169 , piece_id=DSQJ42GPD4PLRLV3OWLD3O3RR5YPPKNWFCLMPE23XRTFBT4APALQ, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 159.223.143.39:49610->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243153239 (5m47.024086616s) [noise=true , address=104.36.82.11:4118 , piece_id=GVKJKH5GFRGVFOV7WKBQXQLJLDSDBKJG4GLYHEQ6YBDI45AH6VVQ, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:104.36.82.11:4118): protocol: write tcp 159.223.143.39:55656->104.36.82.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243308377 (5m47.024241757s) [noise=true , address=104.36.82.55:4228 , piece_id=CJRXNCJOXEMFUHNDOFZEEP6UME67ZUSUHTWR6JDXU3DW22FBR63Q, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 159.223.143.39:52760->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243403740 (5m47.024337118s) [noise=true , address=104.36.82.11:4133 , piece_id=NXAZFSICGF3N3KRVNFI7O6BLPLXV4HXXLRZB45DO4EIYPCZ7MWSQ, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op Jul 26 21:57:31.243509720 (5m47.024443104s) [noise=true , address=104.36.82.42:4798 , piece_id=5VZTWOKNQE64CEVYZ64D2Z4FRKS4CPWPYTJPZTY62BFE74NEPC3A, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 159.223.143.39:35466->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243635732 (5m47.024569111s) [noise=true , address=104.36.82.52:4096 , piece_id=H2KK43MRWWADOPPWG655WQN45CYYP4W6SJVFBWDPLBDGJOG4C6RQ, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: write tcp 159.223.143.39:34614->104.36.82.52:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243708627 (5m47.024641994s) [noise=true , address=104.36.82.53:4142 , piece_id=UWMEWVYCKE53DJ5IBGZHQYU4JQRUAW72QJVGH2AYVQNUB5Y22VNA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 21:57:31.243751703 (5m47.024685066s) [noise=true , address=104.36.82.56:4230 , piece_id=WKUQHC3I6S2D564EYCGSMN5QTGT3B2EMTGVQAROQ2NOEQAIBIF5Q, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: write tcp 159.223.143.39:43992->104.36.82.56:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243799418 (5m47.024732795s) [noise=true , address=104.36.82.47:4542 , piece_id=IPQK4AZXRERR2LC4F4EYGJKRFFTNO6EY4XU3W5DDFVU3ZQAQCHMA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:57:31.243878920 (5m47.024812296s) [noise=true , address=104.36.82.46:4505 , piece_id=CMS2W2HEXZNUOQRTYRXWEILOF7CCHM74MYUCSYUVTS4C76BTFMYA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: rpc: dial tcp 104.36.82.46:4505: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:31.243913165 (5m47.024846545s) [noise=true , address=104.36.82.13:4191 , piece_id=55E5CZ54YW5CDKXV76VAAJB5LH3OYJWFTV5OKIX7EMCX72MLHNOA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:104.36.82.13:4191): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:57:31.243941244 (5m47.02487461s) [noise=true , address=104.36.82.12:4170 , piece_id=VVBD45K2DZGLE5XHTVUJVT6QY6NUDHCRSNPNFPDMWDNC7HYRXOLA, seg_pos={0 0}, upload=74]: Done uploading piece. err: ecclient: upload failed (node:1EfVDsPfsWymV1yfufQw2BidpCGqqqsmtrvoid67a5ug9YYyyd, address:104.36.82.12:4170): protocol: rpc: dial tcp 104.36.82.12:4170: i/o timeout; context deadline exceeded Jul 26 21:57:31.243959775 (5m47.024893158s) [seg_pos={0 0}, upload=74]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 21:57:31.272849859 (5m47.053783242s) [upload=74]: Done uploading segments... Jul 26 21:57:31.272883786 (5m47.053817168s) [upload=74]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.032963047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 13.701770954s FAIL (2025-07-26 21:57:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.701801126s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 10.353136097s FAIL (2025-07-26 21:57:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 2): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037373424 (3m35.818306814s) [noise=true , address=104.36.82.13:4186 , piece_id=3P4TVOZ2BRS2RJT2C6HWXK2ZDN5ZCHAQPWXSMPOD63653PUAZOOA, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037396511 (3m35.818329895s) [noise=true , address=104.36.82.14:4223 , piece_id=4F6ADV22WOQSN3HKFM4JCVZA3OWSYX2S6OP2TAH4SBOKHJ2AQCYQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:51250->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037425149 (3m35.818358546s) [seg_pos={0 0}, upload=43]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:55:20.065111780 (3m35.846045157s) [upload=43]: Done uploading segments... Jul 26 21:55:20.065145366 (3m35.846078742s) [upload=43]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:55:20.164756852 (3m35.945690235s) [upload=50]: Starting upload Jul 26 21:55:20.164893006 (3m35.94582639s) [upload=50]: Done starting upload Jul 26 21:55:20.165026534 (3m35.945959914s) [upload=50]: Uploading segments... Jul 26 21:55:20.165348430 (3m35.946281792s) [upload=50]: Got next segment. Inline: false Jul 26 21:55:20.165436135 (3m35.946369521s) [upload=50]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:55:20.165462754 (3m35.946396131s) [upload=50]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:55:20.199323680 (3m35.980257066s) [seg_pos={0 0}, upload=50]: Begin upload segment... Jul 26 21:55:20.199390208 (3m35.980323572s) [seg_pos={0 0}, upload=50]: Done begin upload segment. Jul 26 21:55:20.199432997 (3m35.980366388s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199457515 (3m35.980390901s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199933992 (3m35.98086738s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200044523 (3m35.98097791s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200436225 (3m35.981369612s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200695832 (3m35.981629215s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200822101 (3m35.98175549s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201228239 (3m35.982161626s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201625940 (3m35.982559326s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201947856 (3m35.982881236s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202017752 (3m35.982951149s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202423768 (3m35.983357154s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202810152 (3m35.983743548s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203207751 (3m35.984141134s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203572200 (3m35.984505585s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203919318 (3m35.984852704s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.305395509 (3m36.086328891s) [upload=50]: Next returned nil segment Jul 26 21:55:20.305415668 (3m36.08634905s) [upload=50]: Waiting for error group managing segments... Jul 26 21:55:20.312143340 (3m36.093076718s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313065385 (3m36.093998763s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313873574 (3m36.094806954s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313911757 (3m36.094845139s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314560119 (3m36.095493488s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314625417 (3m36.095558796s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315101138 (3m36.096034506s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315321282 (3m36.096254661s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315702554 (3m36.096635934s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315909249 (3m36.096842626s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.353897443 (3m41.134830808s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.406007691 (3m41.186941069s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:50.132368988 (4m5.913302372s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132475169 (4m5.913408545s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132612953 (4m5.913546337s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132844023 (4m5.913777387s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132873238 (4m5.913806624s) [seg_pos={0 0}, upload=50]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:55:50.156981397 (4m5.937914767s) [upload=50]: Done uploading segments... Jul 26 21:55:50.157007415 (4m5.93794079s) [upload=50]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.353165916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.031543912s FAIL (2025-07-26 21:57:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031574365s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)