Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 362834 reports (Page 6763 of 7257)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...........] duration: 1.840566776s FAIL (2025-07-26 23:46:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.840591407s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 30.027787578s FAIL (2025-07-26 23:46:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:45:47.528550027 (1h54m3.309483396s) [upload=1747]: Starting upload Jul 26 23:45:47.528640839 (1h54m3.309574218s) [upload=1747]: Done starting upload Jul 26 23:45:47.528745650 (1h54m3.309679015s) [upload=1747]: Uploading segments... Jul 26 23:45:47.529023904 (1h54m3.309957269s) [upload=1747]: Got next segment. Inline: false Jul 26 23:45:47.529113546 (1h54m3.310046926s) [upload=1747]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:47.529121441 (1h54m3.31005482s) [upload=1747]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:47.558220849 (1h54m3.339154215s) [seg_pos={0 0}, upload=1747]: Begin upload segment... Jul 26 23:45:47.558283441 (1h54m3.33921682s) [seg_pos={0 0}, upload=1747]: Done begin upload segment. Jul 26 23:45:47.558351476 (1h54m3.339284867s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558369352 (1h54m3.339302731s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558827957 (1h54m3.339761335s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558903236 (1h54m3.33983662s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559214896 (1h54m3.34014828s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559661333 (1h54m3.340594713s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559999040 (1h54m3.340932417s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560036872 (1h54m3.340970257s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560402073 (1h54m3.341335454s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560498278 (1h54m3.341431655s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560707268 (1h54m3.341640635s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560776189 (1h54m3.341709568s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561074124 (1h54m3.342007503s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561367507 (1h54m3.342300882s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561671774 (1h54m3.342605152s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561996146 (1h54m3.342929522s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.649771810 (1h54m3.430705175s) [upload=1747]: Next returned nil segment Jul 26 23:45:47.649792842 (1h54m3.430726211s) [upload=1747]: Waiting for error group managing segments... Jul 26 23:45:47.653651879 (1h54m3.434585247s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.655490460 (1h54m3.436423829s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.656316568 (1h54m3.437249937s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.658051626 (1h54m3.438984993s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.663908347 (1h54m3.444841723s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.889974750 (1h54m3.67090812s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:49.022796696 (1h54m4.803730077s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:49.105919749 (1h54m4.886853117s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:46:17.498482600 (1h54m33.279415977s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498555419 (1h54m33.279488797s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498588278 (1h54m33.279521641s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498659084 (1h54m33.27959247s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498675455 (1h54m33.279608817s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:47826->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498689802 (1h54m33.279623172s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498733751 (1h54m33.279667257s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498783558 (1h54m33.279716935s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:33490->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498804857 (1h54m33.279738224s) [seg_pos={0 0}, upload=1747]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:46:17.524551936 (1h54m33.305485313s) [upload=1747]: Done uploading segments... Jul 26 23:46:17.524598794 (1h54m33.30553217s) [upload=1747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.027876714s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..........] duration: 2.834682599s FAIL (2025-07-26 23:46:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.834709429s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX...X...X..XX.X...X..X.......X...X....XX..............X.X.] duration: 19.422835539s FAIL (2025-07-26 23:46:00 UTC) context deadline exceeded Duration: 19.422862753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034225081s FAIL (2025-07-26 23:45:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.034250318s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..XX...X...X..XX.X...X..X.......X...X....XX..............X.X] duration: 1m0.001079875s FAIL (2025-07-26 23:45:41 UTC) context deadline exceeded Duration: 1m0.001168071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03371062s FAIL (2025-07-26 23:45:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=1726]: Done begin upload segment. Jul 26 23:44:35.488979313 (1h52m51.269912699s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489001662 (1h52m51.269935044s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489541468 (1h52m51.270474856s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489575066 (1h52m51.270508451s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490050535 (1h52m51.270983919s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490125109 (1h52m51.271058494s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490686708 (1h52m51.271620098s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490794891 (1h52m51.271728282s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.491316121 (1h52m51.272249506s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.491691604 (1h52m51.272624987s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.492241107 (1h52m51.273174493s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.492725186 (1h52m51.273658573s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.493214294 (1h52m51.274147682s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.494016913 (1h52m51.274950301s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.494443652 (1h52m51.275377043s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495095329 (1h52m51.276028718s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495535278 (1h52m51.276468675s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495982709 (1h52m51.276916088s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.565482117 (1h52m51.346415485s) [upload=1726]: Next returned nil segment Jul 26 23:44:35.565499405 (1h52m51.346432784s) [upload=1726]: Waiting for error group managing segments... Jul 26 23:44:35.570357164 (1h52m51.351290541s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.589069819 (1h52m51.370003196s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.615620249 (1h52m51.396553628s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.666362835 (1h52m51.447296203s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.843127855 (1h52m51.624061235s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 26 23:44:36.095341527 (1h52m51.876274889s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:36.619796257 (1h52m52.400729624s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:44:37.623736203 (1h52m53.404669582s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:38.346038855 (1h52m54.126972234s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:38.748590410 (1h52m54.529523779s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:39.757730680 (1h52m55.538664049s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:45:35.421284307 (1h53m51.202217675s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 159.223.143.39:45372->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421302414 (1h53m51.202235796s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:45:35.421406438 (1h53m51.202339807s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:45:35.421690535 (1h53m51.202623904s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421736944 (1h53m51.202670307s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421795354 (1h53m51.20272872s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421847011 (1h53m51.202780377s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421871433 (1h53m51.202804809s) [seg_pos={0 0}, upload=1726]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:45:35.451883786 (1h53m51.232817154s) [upload=1726]: Done uploading segments... Jul 26 23:45:35.451947357 (1h53m51.232880736s) [upload=1726]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.033817297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [.....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032771051s FAIL (2025-07-26 23:45:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032795171s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 30.033294463s FAIL (2025-07-26 23:45:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 0552 (1h52m54.191933919s) [upload=1728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:44:38.411008360 (1h52m54.191941737s) [upload=1728]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:44:38.447363024 (1h52m54.228296409s) [seg_pos={0 0}, upload=1728]: Begin upload segment... Jul 26 23:44:38.447442037 (1h52m54.228375417s) [seg_pos={0 0}, upload=1728]: Done begin upload segment. Jul 26 23:44:38.447472505 (1h52m54.228405885s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.447530743 (1h52m54.228464133s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448168457 (1h52m54.229101841s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448218182 (1h52m54.229151566s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448712998 (1h52m54.229646397s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448734552 (1h52m54.229668042s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449124782 (1h52m54.230058176s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449223824 (1h52m54.230157203s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449621031 (1h52m54.23055442s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449737624 (1h52m54.23067101s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449998923 (1h52m54.230932312s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.450229755 (1h52m54.231163142s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.450770002 (1h52m54.23170343s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451012447 (1h52m54.23194584s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451269660 (1h52m54.232203049s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451502954 (1h52m54.232436341s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.561793425 (1h52m54.342726809s) [upload=1728]: Next returned nil segment Jul 26 23:44:38.561815173 (1h52m54.34274856s) [upload=1728]: Waiting for error group managing segments... Jul 26 23:44:38.570459813 (1h52m54.351393192s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571277426 (1h52m54.352210805s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571419049 (1h52m54.352352426s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571956093 (1h52m54.352889472s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.572036717 (1h52m54.352970086s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.572566661 (1h52m54.353500038s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:45:08.379149593 (1h53m24.16008296s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379256562 (1h53m24.160189944s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379323343 (1h53m24.160256722s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379393923 (1h53m24.160327286s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46344->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379446117 (1h53m24.160379495s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379515407 (1h53m24.160448787s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:49236->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379578632 (1h53m24.160511996s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379653427 (1h53m24.160586806s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379668067 (1h53m24.160601453s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379736540 (1h53m24.160669921s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379804090 (1h53m24.160737483s) [seg_pos={0 0}, upload=1728]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:45:08.410118785 (1h53m24.191052163s) [upload=1728]: Done uploading segments... Jul 26 23:45:08.410149846 (1h53m24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.033321456s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.........] duration: 2.641604534s FAIL (2025-07-26 23:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.641768951s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX........] duration: 4.863295244s FAIL (2025-07-26 23:44:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.863320337s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [X.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030629469s FAIL (2025-07-26 23:44:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ZHPJXGXZTZUIULDOD2DK7ISAEB5NQWW6Q7Z3IF7Q6GTLT6AYSQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.391716684 (1h51m51.172650059s) [noise=true , address=77.90.31.11:4579 , piece_id=PGGZXX3UF4YRHNGSHBPWAC3KC6XP3OP7N3HP5BLEY3A4XFBVIUHA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392142947 (1h51m51.173076325s) [noise=true , address=77.90.31.15:4722 , piece_id=2ZFNJBPL5E6RQIGHTZDAHXA665ZP3WSFJUD4WZO72WJTFZNM3N4A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392207849 (1h51m51.173141233s) [noise=true , address=77.90.31.19:4871 , piece_id=JNLQDO3KD3ESRGHTHC3VQHASJECGGJ33IN5N5SI7KGXXUSQ6A47A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392518074 (1h51m51.173451449s) [noise=true , address=104.36.82.55:4218 , piece_id=ON3FC4E7PWTMO65YKFBWRKHJ6DCDEUYKGUMGVDXA35SHA3ECM5WA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392555483 (1h51m51.173488869s) [noise=true , address=104.36.82.46:4504 , piece_id=MHI5OCHIDHTFKYW7WOSOOW6II2DWP6TSH3RVJ4H52UWT3SPWKWLA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392790318 (1h51m51.173723696s) [noise=true , address=77.90.31.21:4912 , piece_id=IG4UTY33ZJWL4FAFTLO4V72KMAUMOYARYZKOTH7JWKLNCW7J43GA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.393129281 (1h51m51.174062655s) [noise=true , address=104.36.82.11:4131 , piece_id=4YUWTYDBI6JY5XHAD4TFORRWO565FIN2UCK54YOFWZMCQEMKXGIQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.395679464 (1h51m51.176612849s) [noise=true , address=79.127.219.38:4009 , piece_id=PI3QKMMURZP4UPSTJVYSVLLPLPMB7LAL4LOVI7KJVMO3JFRIPWKQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.397122308 (1h51m51.178055693s) [noise=true , address=104.36.82.56:4245 , piece_id=5WXBJXSZ2W3YSJIPEEA2X4AP5QF5ABPS2M2UEHF7W3EQICMGARDA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.398564287 (1h51m51.179497675s) [noise=true , address=64.49.13.137:4502 , piece_id=BLP7EUD6IOLINXDNIQDONHZUS5ZPQYPBGYJZMVER7QE7IJJIDDUQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.399672121 (1h51m51.180605503s) [noise=true , address=79.127.205.233:4642 , piece_id=BV3ICMTEQXZFE4NQ2O5ESRCAIV6G7BV2YQ5F4WIHA4IO47I425MQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.400452722 (1h51m51.181386098s) [noise=true , address=79.127.220.101:4041 , piece_id=562KM7QRHXSAE4O2A4I2HEZEUJQGD63J4AFEDJDQOEYKIZFLYZ2A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.401028982 (1h51m51.181962343s) [noise=true , address=79.127.201.215:4357 , piece_id=YUCRCTBNNTUZB46AHTWVWCW32N5ANN2FWMGFLMYETZTWJXGLRIJA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.401558656 (1h51m51.182492043s) [noise=true , address=77.90.31.16:4765 , piece_id=ODOACHSC67Q3XGEBQBDGIJE5UXWFYUPQNCGR7CLJU5FGFFGA4FKA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.402137293 (1h51m51.183070679s) [noise=true , address=104.36.82.53:4152 , piece_id=5YY4SRMO5G4MOHFEFYRT35CS3ZII35UZ2HLOUWBNFQVCFR2XYFGQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.402671349 (1h51m51.183604737s) [noise=true , address=104.36.82.47:4535 , piece_id=WYVLKBWVMCPBA2XNH6EBBZJQJ2OWGWUPDC5KNN2MVI4F3RUFDTWQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.403312128 (1h51m51.184245514s) [noise=true , address=79.127.201.214:4321 , piece_id=57H6EJTATVVHPTPVSU4ZGJM3YW6KFRBFDFCN4V3HHBXYDHSHKJHQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.452911857 (1h51m51.233845233s) [upload=1713]: Next returned nil segment Jul 26 23:43:35.452931776 (1h51m51.233865162s) [upload=1713]: Waiting for error group managing segments... Jul 26 23:43:35.479860829 (1h51m51.260794205s) [noise=true , address=77.90.31.11:4579 , piece_id=PGGZXX3UF4YRHNGSHBPWAC3KC6XP3OP7N3HP5BLEY3A4XFBVIUHA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:35.499783881 (1h51m51.280717248s) [noise=true , address=79.127.219.38:4009 , piece_id=PI3QKMMURZP4UPSTJVYSVLLPLPMB7LAL4LOVI7KJVMO3JFRIPWKQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:35.550152358 (1h51m51.331085733s) [noise=true , address=79.127.201.214:4321 , piece_id=57H6EJTATVVHPTPVSU4ZGJM3YW6KFRBFDFCN4V3HHBXYDHSHKJHQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:35.969806259 (1h51m51.750739873s) [noise=true , address=79.127.205.233:4642 , piece_id=BV3ICMTEQXZFE4NQ2O5ESRCAIV6G7BV2YQ5F4WIHA4IO47I425MQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:36.019191602 (1h51m51.800124986s) [noise=true , address=79.127.205.245:4236 , piece_id=4NZHPJXGXZTZUIULDOD2DK7ISAEB5NQWW6Q7Z3IF7Q6GTLT6AYSQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:39.604338610 (1h51m55.385271992s) [noise=true , address=77.90.31.19:4871 , piece_id=JNLQDO3KD3ESRGHTHC3VQHASJECGGJ33IN5N5SI7KGXXUSQ6A47A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:39.927525895 (1h51m55.708459263s) [noise=true , address=79.127.201.215:4357 , piece_id=YUCRCTBNNTUZB46AHTWVWCW32N5ANN2FWMGFLMYETZTWJXGLRIJA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:41.343440817 (1h51m57.124374198s) [noise=true , address=79.127.220.101:4041 , piece_id=562KM7QRHXSAE4O2A4I2HEZEUJQGD63J4AFEDJDQOEYKIZFLYZ2A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:41.876631082 (1h51m57.657564469s) [noise=true , address=64.49.13.137:4502 , piece_id=BLP7EUD6IOLINXDNIQDONHZUS5ZPQYPBGYJZMVER7QE7IJJIDDUQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:53.156195743 (1h52m8.937129124s) [noise=true , address=77.90.31.21:4912 , piece_id=IG4UTY33ZJWL4FAFTLO4V72KMAUMOYARYZKOTH7JWKLNCW7J43GA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: read tcp 159.223.143.39:54224->77.90.31.21:4912: read: connection reset by peer; conn closed: op Jul 26 23:44:16.393028126 (1h52m32.173961493s) [noise=true , address=77.90.31.15:4722 , piece_id=2ZFNJBPL5E6RQIGHTZDAHXA665ZP3WSFJUD4WZO72WJTFZNM3N4A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:44:35.326069443 (1h52m51.107002823s) [noise=true , address=104.36.82.11:4131 , piece_id=4YUWTYDBI6JY5XHAD4TFORRWO565FIN2UCK54YOFWZMCQEMKXGIQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: rpc: dial tcp 104.36.82.11:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326142016 (1h52m51.107075392s) [noise=true , address=104.36.82.47:4535 , piece_id=WYVLKBWVMCPBA2XNH6EBBZJQJ2OWGWUPDC5KNN2MVI4F3RUFDTWQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326212147 (1h52m51.107145524s) [noise=true , address=104.36.82.46:4504 , piece_id=MHI5OCHIDHTFKYW7WOSOOW6II2DWP6TSH3RVJ4H52UWT3SPWKWLA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326282132 (1h52m51.107215508s) [noise=true , address=77.90.31.16:4765 , piece_id=ODOACHSC67Q3XGEBQBDGIJE5UXWFYUPQNCGR7CLJU5FGFFGA4FKA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: read tcp 159.223.143.39:49458->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326290240 (1h52m51.107223607s) [noise=true , address=104.36.82.56:4245 , piece_id=5WXBJXSZ2W3YSJIPEEA2X4AP5QF5ABPS2M2UEHF7W3EQICMGARDA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: rpc: dial tcp 104.36.82.56:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326298451 (1h52m51.10723182s) [noise=true , address=104.36.82.55:4218 , piece_id=ON3FC4E7PWTMO65YKFBWRKHJ6DCDEUYKGUMGVDXA35SHA3ECM5WA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326336204 (1h52m51.10726958s) [noise=true , address=104.36.82.53:4152 , piece_id=5YY4SRMO5G4MOHFEFYRT35CS3ZII35UZ2HLOUWBNFQVCFR2XYFGQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: rpc: dial tcp 104.36.82.53:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326363682 (1h52m51.10729706s) [seg_pos={0 0}, upload=1713]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:44:35.354528253 (1h52m51.135461628s) [upload=1713]: Done uploading segments... Jul 26 23:44:35.354583407 (1h52m51.135516784s) [upload=1713]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.030654169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [......X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034044668s FAIL (2025-07-26 23:44:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.034066538s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.036518242s FAIL (2025-07-26 23:44:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== :43:54.157765259 (1h52m9.938698635s) [upload=1718]: Done starting upload Jul 26 23:43:54.157859239 (1h52m9.938792605s) [upload=1718]: Uploading segments... Jul 26 23:43:54.158353784 (1h52m9.939287161s) [upload=1718]: Got next segment. Inline: false Jul 26 23:43:54.158396626 (1h52m9.939330002s) [upload=1718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:54.158403266 (1h52m9.939336643s) [upload=1718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:54.270125362 (1h52m10.051058751s) [seg_pos={0 0}, upload=1718]: Begin upload segment... Jul 26 23:43:54.270192378 (1h52m10.051125767s) [seg_pos={0 0}, upload=1718]: Done begin upload segment. Jul 26 23:43:54.270246643 (1h52m10.051180023s) [noise=true , address=104.36.82.6:4874 , piece_id=D3T765DB4TZ2P3M7X5NJPFTUUKJTFHSQQOPK7DQSHGXYGMAWI4UQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270259443 (1h52m10.05119283s) [noise=true , address=104.36.82.2:4731 , piece_id=5MYU2ZPKYWTBBZNDRQ64VX7VUIDJ2CTVB7RKLFLEIEKIYF7TDTPA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270717120 (1h52m10.051650504s) [noise=true , address=45.91.116.6:4973 , piece_id=OEIDGH5JAOSTKOUS3Y5SSJKRIHK4D6RZX37NFMWJVGKWRZ75GGQA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270794603 (1h52m10.051727983s) [noise=true , address=77.90.31.16:4734 , piece_id=QHJ37CVOP3IFGQUUEZWSYFMD2ZAZLF3T7CUV4XI4RHOJUXW35CEQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271154154 (1h52m10.05208754s) [noise=true , address=45.91.116.4:4937 , piece_id=H6DG5P7GOZ44G6MJNVT4UJ4ZJXPGFUBOWLRGGH3YDIAIYT44FY5Q, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271231647 (1h52m10.052165015s) [noise=true , address=45.91.116.8:4047 , piece_id=O7ZWWN5G6AOLUJYZLM3L3QTZ4AI5VHND33XITRPIWN5T6WRSGHMA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271543638 (1h52m10.052477024s) [noise=true , address=104.36.82.5:4839 , piece_id=JTG3HD3JALUU3C5TYG7BXNTMM56LSQ3HZQFN5NRU5EVES4VRUXSA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271903071 (1h52m10.052836456s) [noise=true , address=77.90.31.18:4806 , piece_id=UBFJV4OQLHH65H7MRPNRGO5AK5F4YV25K2EQCVKMT6ZTM7MX46HQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272355961 (1h52m10.05328934s) [noise=true , address=104.36.82.16:4293 , piece_id=N4L3ZUWX6CXAEKBJQHYAFALP7QM7PYM467PFU7Y2YAC6WENBIVMA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272735948 (1h52m10.053669333s) [noise=true , address=104.36.82.12:4151 , piece_id=EXYORV37DH4P7FCRQILTM3Q2FVG2BTHZXN3RM2B6PEQHUISZ6KPQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272801587 (1h52m10.053734965s) [noise=true , address=45.91.116.9:4263 , piece_id=6UBRIDEXLIKU43HCQFCHFZH34HOGLYUE76UGSVCDFWYKLYRSH4BA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273177555 (1h52m10.054110932s) [noise=true , address=104.36.82.13:4187 , piece_id=C3CXAWPGYAZ6RCTJXIQLJSNMRMSH2XGHR2RSPL3OQTT6AMHQKFNA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273500189 (1h52m10.054433553s) [noise=true , address=77.90.31.14:4663 , piece_id=TP7YWEMBTCDJEXO3U54IK2645BVHGGLQYKCIUVVON6YLKV3UHX7A, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273775215 (1h52m10.05470859s) [noise=true , address=104.36.82.5:4838 , piece_id=VLMXD4IX5XLQZ2627JRMHBKEQA5YWPQC3BDMCH277SMHCXWNHJWA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.274072283 (1h52m10.055005701s) [noise=true , address=45.91.116.13:4155 , piece_id=66FX6BAFHG7IYZMIBLR4W7PDDW5YGTJQUOUF7RLYWONYGNUWTVHA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.274141720 (1h52m10.05507511s) [noise=true , address=104.36.82.4:4802 , piece_id=VTTI6JL6AMHJBSUEXGAYSIPUTJQELYQYYNSORDH6QIP4AKAY4HBA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.345162001 (1h52m10.126095391s) [upload=1718]: Next returned nil segment Jul 26 23:43:54.345182867 (1h52m10.126116245s) [upload=1718]: Waiting for error group managing segments... Jul 26 23:43:54.349720399 (1h52m10.130653764s) [noise=true , address=45.91.116.13:4155 , piece_id=66FX6BAFHG7IYZMIBLR4W7PDDW5YGTJQUOUF7RLYWONYGNUWTVHA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.349743570 (1h52m10.130676952s) [noise=true , address=45.91.116.6:4973 , piece_id=OEIDGH5JAOSTKOUS3Y5SSJKRIHK4D6RZX37NFMWJVGKWRZ75GGQA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350301115 (1h52m10.13123448s) [noise=true , address=45.91.116.8:4047 , piece_id=O7ZWWN5G6AOLUJYZLM3L3QTZ4AI5VHND33XITRPIWN5T6WRSGHMA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350324283 (1h52m10.131257652s) [noise=true , address=45.91.116.4:4937 , piece_id=H6DG5P7GOZ44G6MJNVT4UJ4ZJXPGFUBOWLRGGH3YDIAIYT44FY5Q, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350735883 (1h52m10.131669249s) [noise=true , address=77.90.31.14:4663 , piece_id=TP7YWEMBTCDJEXO3U54IK2645BVHGGLQYKCIUVVON6YLKV3UHX7A, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350749964 (1h52m10.131683325s) [noise=true , address=77.90.31.18:4806 , piece_id=UBFJV4OQLHH65H7MRPNRGO5AK5F4YV25K2EQCVKMT6ZTM7MX46HQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.462953503 (1h52m10.243886864s) [noise=true , address=45.91.116.9:4263 , piece_id=6UBRIDEXLIKU43HCQFCHFZH34HOGLYUE76UGSVCDFWYKLYRSH4BA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:44:24.119156911 (1h52m39.900090292s) [noise=true , address=77.90.31.16:4734 , piece_id=QHJ37CVOP3IFGQUUEZWSYFMD2ZAZLF3T7CUV4XI4RHOJUXW35CEQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119243688 (1h52m39.900177076s) [noise=true , address=104.36.82.13:4187 , piece_id=C3CXAWPGYAZ6RCTJXIQLJSNMRMSH2XGHR2RSPL3OQTT6AMHQKFNA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119314962 (1h52m39.900248342s) [noise=true , address=104.36.82.12:4151 , piece_id=EXYORV37DH4P7FCRQILTM3Q2FVG2BTHZXN3RM2B6PEQHUISZ6KPQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119387866 (1h52m39.900321246s) [noise=true , address=104.36.82.5:4839 , piece_id=JTG3HD3JALUU3C5TYG7BXNTMM56LSQ3HZQFN5NRU5EVES4VRUXSA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119400276 (1h52m39.90033365s) [noise=true , address=104.36.82.2:4731 , piece_id=5MYU2ZPKYWTBBZNDRQ64VX7VUIDJ2CTVB7RKLFLEIEKIYF7TDTPA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119573282 (1h52m39.900506676s) [noise=true , address=104.36.82.6:4874 , piece_id=D3T765DB4TZ2P3M7X5NJPFTUUKJTFHSQQOPK7DQSHGXYGMAWI4UQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119645065 (1h52m39.900578446s) [noise=true , address=104.36.82.5:4838 , piece_id=VLMXD4IX5XLQZ2627JRMHBKEQA5YWPQC3BDMCH277SMHCXWNHJWA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119722126 (1h52m39.900655508s) [noise=true , address=104.36.82.4:4802 , piece_id=VTTI6JL6AMHJBSUEXGAYSIPUTJQELYQYYNSORDH6QIP4AKAY4HBA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:52390->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119755960 (1h52m39.900689343s) [noise=true , address=104.36.82.16:4293 , piece_id=N4L3ZUWX6CXAEKBJQHYAFALP7QM7PYM467PFU7Y2YAC6WENBIVMA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119816196 (1h52m39.900749575s) [seg_pos={0 0}, upload=1718]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:44:24.152763548 (1h52m39.933696917s) [upload=1718]: Done uploading segments... Jul 26 23:44:24.152787415 (1h52m39.933720792s) [upload=1718]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.03653901s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...XX...X...X..XX.X...X..X.......X...X....XX..............X.] duration: 29.181859737s FAIL (2025-07-26 23:44:24 UTC) context deadline exceeded Duration: 29.181876506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.......] duration: 2.828121224s FAIL (2025-07-26 23:44:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828146534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [.......X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031747019s FAIL (2025-07-26 23:44:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031779289s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....XX...X...X..XX.X...X..X.......X...X....XX..............X] duration: 1m0.001520363s FAIL (2025-07-26 23:43:55 UTC) context deadline exceeded Duration: 1m0.001546332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.027633709s FAIL (2025-07-26 23:43:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 4s) [upload=1710]: Starting upload Jul 26 23:43:24.053285638 (1h51m39.834219013s) [upload=1710]: Done starting upload Jul 26 23:43:24.053401065 (1h51m39.834334428s) [upload=1710]: Uploading segments... Jul 26 23:43:24.053530822 (1h51m39.83446419s) [upload=1710]: Got next segment. Inline: false Jul 26 23:43:24.053574567 (1h51m39.834507942s) [upload=1710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:24.053580169 (1h51m39.834513544s) [upload=1710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:24.085282189 (1h51m39.866215564s) [seg_pos={0 0}, upload=1710]: Begin upload segment... Jul 26 23:43:24.085328954 (1h51m39.86626232s) [seg_pos={0 0}, upload=1710]: Done begin upload segment. Jul 26 23:43:24.085367234 (1h51m39.866300615s) [noise=true , address=104.36.82.3:4765 , piece_id=SJXRFQ3BCN4SSNERWLVXKR7D2OHIILNRIL3YYFINTZQHMU5K2FHQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.086026246 (1h51m39.866959633s) [noise=true , address=104.36.82.13:4187 , piece_id=MGSDRD2W4BPIQN2LFRMOWRNSOFMHBD46BFIWILZD6OE4FZHFL75Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.086080520 (1h51m39.867013904s) [noise=true , address=45.91.116.10:4082 , piece_id=6Q2ZG5QWFBWA7MZEXVW7CAGYAKZ2JHXD6ZGCAMPRA3PPEDYOPE4A, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.087104314 (1h51m39.868037698s) [noise=true , address=104.36.82.12:4151 , piece_id=O7VUC3F54EUMNWSQJW62UCKFOJUTBTQGSJ7JEMI3VH5IRBNZ7DJA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.087162462 (1h51m39.868095847s) [noise=true , address=45.91.116.7:4010 , piece_id=XMJL46CSVMHHNLFKSMSTAVFSKMO5XIEGTUEBGT47HPEO7NLOBAIA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.088345958 (1h51m39.869279337s) [noise=true , address=104.36.82.8:4970 , piece_id=M6T2NAYIB2DW7WI5W6INDUJ7WYJRZ4OWCVQKX6ZTMADOGYVX7ZEQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.088586179 (1h51m39.869519561s) [noise=true , address=104.36.82.7:4911 , piece_id=ZZYBMRSP3EDB656PLBHPO5XVIZBEVBJTGCC66CRTOHH5WXX5O52Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089178665 (1h51m39.8701121s) [noise=true , address=45.91.116.5:4297 , piece_id=WPPROWUKTRZIOFX77EFGHTBVIMHQVTWURA5ZURFXDSNEIXRZ47RA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089233624 (1h51m39.870167007s) [noise=true , address=45.91.116.9:4261 , piece_id=SGHKYPAS5DRZXTXNN2WV3BEUKR5V23YNF4CU23BZBXDMJIIMP5SQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089707629 (1h51m39.870641017s) [noise=true , address=104.36.82.6:4875 , piece_id=XDS7O46ZYGEW4ICUKHUBU4JPLKE5LBJ5ZHKDH74R6TDKPG3VXMLQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089760611 (1h51m39.87069399s) [noise=true , address=104.36.82.2:4730 , piece_id=DFAMBBZWDG66X437Z3C4IFSVB7RA5VSOMJI266WPFEQDEFPJOGTQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090148676 (1h51m39.871082059s) [noise=true , address=104.36.82.12:4150 , piece_id=KNDZDHDXLDTZKULKDPUTPU3XERXIMIQIWI374ZJGMTG7KWJZDGDA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090338084 (1h51m39.871271464s) [noise=true , address=77.90.31.14:4661 , piece_id=4V4SKGXUKFW5OCXNXO2HGIXJERAQRLHLFEWGV47JEV6DLNBJ6AOQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090583734 (1h51m39.871517111s) [noise=true , address=104.36.82.10:4079 , piece_id=ZVQ2CEJWNEBJ7ZWNCK4C7PRJ4L7SATME3L2ZW3NEYFM3SNCGTDFQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090979454 (1h51m39.871912835s) [noise=true , address=45.91.116.12:4226 , piece_id=LWEDERCKIHG7A4CKMILFZPEY7BDP3UXHASHAZHQUHGKPHM3MCQRA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.091435268 (1h51m39.872368653s) [noise=true , address=45.91.116.13:4155 , piece_id=HJUWMD4G5CGV2I6JUTGBOFQRYC3PBC3YHMC4WYKL662HZA74I55Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.190964403 (1h51m39.97189778s) [upload=1710]: Next returned nil segment Jul 26 23:43:24.190982901 (1h51m39.97191628s) [upload=1710]: Waiting for error group managing segments... Jul 26 23:43:24.194466242 (1h51m39.975399625s) [noise=true , address=45.91.116.7:4010 , piece_id=XMJL46CSVMHHNLFKSMSTAVFSKMO5XIEGTUEBGT47HPEO7NLOBAIA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.194667599 (1h51m39.97560096s) [noise=true , address=45.91.116.12:4226 , piece_id=LWEDERCKIHG7A4CKMILFZPEY7BDP3UXHASHAZHQUHGKPHM3MCQRA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.194861046 (1h51m39.975794428s) [noise=true , address=45.91.116.10:4082 , piece_id=6Q2ZG5QWFBWA7MZEXVW7CAGYAKZ2JHXD6ZGCAMPRA3PPEDYOPE4A, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.195604317 (1h51m39.976537696s) [noise=true , address=77.90.31.14:4661 , piece_id=4V4SKGXUKFW5OCXNXO2HGIXJERAQRLHLFEWGV47JEV6DLNBJ6AOQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.196035791 (1h51m39.976969167s) [noise=true , address=45.91.116.13:4155 , piece_id=HJUWMD4G5CGV2I6JUTGBOFQRYC3PBC3YHMC4WYKL662HZA74I55Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.271337016 (1h51m40.052270393s) [noise=true , address=45.91.116.9:4261 , piece_id=SGHKYPAS5DRZXTXNN2WV3BEUKR5V23YNF4CU23BZBXDMJIIMP5SQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.608416573 (1h51m40.389349942s) [noise=true , address=45.91.116.5:4297 , piece_id=WPPROWUKTRZIOFX77EFGHTBVIMHQVTWURA5ZURFXDSNEIXRZ47RA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:54.026196177 (1h52m9.807129564s) [noise=true , address=104.36.82.10:4079 , piece_id=ZVQ2CEJWNEBJ7ZWNCK4C7PRJ4L7SATME3L2ZW3NEYFM3SNCGTDFQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026329659 (1h52m9.807263035s) [noise=true , address=104.36.82.13:4187 , piece_id=MGSDRD2W4BPIQN2LFRMOWRNSOFMHBD46BFIWILZD6OE4FZHFL75Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026376716 (1h52m9.807310081s) [noise=true , address=104.36.82.2:4730 , piece_id=DFAMBBZWDG66X437Z3C4IFSVB7RA5VSOMJI266WPFEQDEFPJOGTQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026462281 (1h52m9.807395664s) [noise=true , address=104.36.82.12:4150 , piece_id=KNDZDHDXLDTZKULKDPUTPU3XERXIMIQIWI374ZJGMTG7KWJZDGDA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026564015 (1h52m9.807497395s) [noise=true , address=104.36.82.6:4875 , piece_id=XDS7O46ZYGEW4ICUKHUBU4JPLKE5LBJ5ZHKDH74R6TDKPG3VXMLQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026646507 (1h52m9.807579888s) [noise=true , address=104.36.82.8:4970 , piece_id=M6T2NAYIB2DW7WI5W6INDUJ7WYJRZ4OWCVQKX6ZTMADOGYVX7ZEQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026754706 (1h52m9.807688071s) [noise=true , address=104.36.82.3:4765 , piece_id=SJXRFQ3BCN4SSNERWLVXKR7D2OHIILNRIL3YYFINTZQHMU5K2FHQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026809577 (1h52m9.807742944s) [noise=true , address=104.36.82.12:4151 , piece_id=O7VUC3F54EUMNWSQJW62UCKFOJUTBTQGSJ7JEMI3VH5IRBNZ7DJA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026876891 (1h52m9.807810258s) [noise=true , address=104.36.82.7:4911 , piece_id=ZZYBMRSP3EDB656PLBHPO5XVIZBEVBJTGCC66CRTOHH5WXX5O52Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026904979 (1h52m9.807838356s) [seg_pos={0 0}, upload=1710]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:43:54.051099432 (1h52m9.832032809s) [upload=1710]: Done uploading segments... Jul 26 23:43:54.051131056 (1h52m9.832064436s) [upload=1710]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.027654647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031471146s FAIL (2025-07-26 23:43:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 95]: Begin upload segment... Jul 26 23:42:35.282602134 (1h50m51.063535515s) [seg_pos={0 0}, upload=1695]: Done begin upload segment. Jul 26 23:42:35.282651075 (1h50m51.063584462s) [noise=true , address=77.90.31.18:4812 , piece_id=KYDR5NBFVGYT37SDVW7LIPYDQETTU63YGTBQJGRX25TOYQHDUIFA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.282675926 (1h50m51.06360931s) [noise=true , address=77.90.31.10:4543 , piece_id=DYX2L553ZJKNQGPCW2WLUEMDVHUMCEPASQ4TWW3WJ4ZFWUH7PQNA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283305808 (1h50m51.064239193s) [noise=true , address=79.127.205.245:4223 , piece_id=OWLLV7ALZPFK43ARLQOHS5IODPHYOUVK5KJOI722JYYXK7GF3QHA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283868138 (1h50m51.064801524s) [noise=true , address=104.36.82.52:4107 , piece_id=HAHRLH3QEW4A3ABCPWVUDJTGNLDGBJW7DB3N5V3AGUCQLMXKF3BQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283943769 (1h50m51.064877152s) [noise=true , address=79.127.201.214:4321 , piece_id=2XR23HDME4UFYHOEPTM5ZC3BBECEOLLVCOXXKGNCJQQF7X3BPFGA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284422849 (1h50m51.065356233s) [noise=true , address=79.127.220.101:4041 , piece_id=ZHPWOXFKTP4IKTJCWLBZSGB54PWFCA4W6SYMXUKRROF2D6WZKVEA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284467062 (1h50m51.065400443s) [noise=true , address=77.90.31.15:4717 , piece_id=T54LAZH4S4N5KZUW73N2MTWWYJDHGHAXA2ABYPBZWBIP3NHXQAOA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284790285 (1h50m51.065723665s) [noise=true , address=156.146.43.225:4448 , piece_id=RIPBINT4MALR7FC3RDI2BFUWYK4RZISJNB6HYCJN5X4MST3PCDLQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284946847 (1h50m51.06588023s) [noise=true , address=104.36.82.55:4200 , piece_id=LN5DOA2DAM5GIA5JNMZCCVUUOKKMXKYGRWARBAVLZYCAGREWN6PA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285098024 (1h50m51.066031399s) [noise=true , address=104.36.82.54:4190 , piece_id=WNW7ODGCUGS4VAISGRORFKBJCVGPIOFUSIODYE2MRDKXC6XTOZLA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285385085 (1h50m51.06631846s) [noise=true , address=79.127.205.235:4673 , piece_id=32JJW5WDGDT57IIFQQNJE722QQD7BI2YRNTBSU6KTCTLJF26QGTQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285463416 (1h50m51.066396801s) [noise=true , address=104.36.82.47:4523 , piece_id=TA7226R3LANDX5RMISRCJYJ646C25AFRFCEQ6ZWKTP3E75BMQGFA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285939070 (1h50m51.066872459s) [noise=true , address=64.49.13.138:4482 , piece_id=AWXDQ3TRVC3CNNXLZAE5XJBIYYF5UUCVGVK3PKELVEKZ5TLEJVXA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.286377672 (1h50m51.067311055s) [noise=true , address=79.127.219.38:4004 , piece_id=LVSY6XTTEPS6JUPD7YTMDCKW4DZVICXXRO4OGTELHLVSKVI3GXPQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.286802945 (1h50m51.067736327s) [noise=true , address=79.127.201.215:4344 , piece_id=S326D47EJWUNM4TWSEBQ6VXCIVDFLWKMYX46AVPHD6RI4G5BPROQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.287257627 (1h50m51.068191007s) [noise=true , address=79.127.205.234:4610 , piece_id=FHUTQB5EJRF6V6TWIWGVQHFJEQBJWSWOGP7A7OMHBYJZGQPIYFTA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.287665206 (1h50m51.068598593s) [noise=true , address=104.36.82.46:4510 , piece_id=E76TEGR5N5YOK3J4U34V4PFCQ37XRHENDUBGWRM2SXPBNUL3PRAQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.298538049 (1h50m51.079471423s) [noise=true , address=79.127.205.233:4624 , piece_id=ZBXGBEELHHHUFTFEOMNG7ZNJ7CUVJBZPEDVACVMP6QJUTH7YHQZQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.352528873 (1h50m51.133462262s) [upload=1695]: Next returned nil segment Jul 26 23:42:35.352550245 (1h50m51.133483628s) [upload=1695]: Waiting for error group managing segments... Jul 26 23:42:35.445801385 (1h50m51.226734764s) [noise=true , address=79.127.201.214:4321 , piece_id=2XR23HDME4UFYHOEPTM5ZC3BBECEOLLVCOXXKGNCJQQF7X3BPFGA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:35.455970865 (1h50m51.236904227s) [noise=true , address=77.90.31.18:4812 , piece_id=KYDR5NBFVGYT37SDVW7LIPYDQETTU63YGTBQJGRX25TOYQHDUIFA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.705136251 (1h50m51.486069628s) [noise=true , address=79.127.219.38:4004 , piece_id=LVSY6XTTEPS6JUPD7YTMDCKW4DZVICXXRO4OGTELHLVSKVI3GXPQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.827472444 (1h50m51.608405827s) [noise=true , address=64.49.13.138:4482 , piece_id=AWXDQ3TRVC3CNNXLZAE5XJBIYYF5UUCVGVK3PKELVEKZ5TLEJVXA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:35.862584489 (1h50m51.643517866s) [noise=true , address=79.127.205.234:4610 , piece_id=FHUTQB5EJRF6V6TWIWGVQHFJEQBJWSWOGP7A7OMHBYJZGQPIYFTA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.880615994 (1h50m51.661549361s) [noise=true , address=79.127.205.245:4223 , piece_id=OWLLV7ALZPFK43ARLQOHS5IODPHYOUVK5KJOI722JYYXK7GF3QHA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.895091485 (1h50m51.676024865s) [noise=true , address=79.127.205.233:4624 , piece_id=ZBXGBEELHHHUFTFEOMNG7ZNJ7CUVJBZPEDVACVMP6QJUTH7YHQZQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.933695124 (1h50m51.71462849s) [noise=true , address=79.127.205.235:4673 , piece_id=32JJW5WDGDT57IIFQQNJE722QQD7BI2YRNTBSU6KTCTLJF26QGTQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:37.280783511 (1h50m53.06171688s) [noise=true , address=79.127.220.101:4041 , piece_id=ZHPWOXFKTP4IKTJCWLBZSGB54PWFCA4W6SYMXUKRROF2D6WZKVEA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:38.141031537 (1h50m53.921964913s) [noise=true , address=156.146.43.225:4448 , piece_id=RIPBINT4MALR7FC3RDI2BFUWYK4RZISJNB6HYCJN5X4MST3PCDLQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:58.856493400 (1h51m14.637426768s) [noise=true , address=79.127.201.215:4344 , piece_id=S326D47EJWUNM4TWSEBQ6VXCIVDFLWKMYX46AVPHD6RI4G5BPROQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:59.354096556 (1h51m15.135029936s) [noise=true , address=77.90.31.15:4717 , piece_id=T54LAZH4S4N5KZUW73N2MTWWYJDHGHAXA2ABYPBZWBIP3NHXQAOA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:43:35.221090919 (1h51m51.002024296s) [noise=true , address=104.36.82.52:4107 , piece_id=HAHRLH3QEW4A3ABCPWVUDJTGNLDGBJW7DB3N5V3AGUCQLMXKF3BQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221147081 (1h51m51.002080457s) [noise=true , address=77.90.31.10:4543 , piece_id=DYX2L553ZJKNQGPCW2WLUEMDVHUMCEPASQ4TWW3WJ4ZFWUH7PQNA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 159.223.143.39:56664->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221183240 (1h51m51.002116617s) [noise=true , address=104.36.82.47:4523 , piece_id=TA7226R3LANDX5RMISRCJYJ646C25AFRFCEQ6ZWKTP3E75BMQGFA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: rpc: dial tcp 104.36.82.47:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221229901 (1h51m51.002163277s) [noise=true , address=104.36.82.46:4510 , piece_id=E76TEGR5N5YOK3J4U34V4PFCQ37XRHENDUBGWRM2SXPBNUL3PRAQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:37314->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221269138 (1h51m51.002202514s) [noise=true , address=104.36.82.55:4200 , piece_id=LN5DOA2DAM5GIA5JNMZCCVUUOKKMXKYGRWARBAVLZYCAGREWN6PA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221299563 (1h51m51.002232924s) [noise=true , address=104.36.82.54:4190 , piece_id=WNW7ODGCUGS4VAISGRORFKBJCVGPIOFUSIODYE2MRDKXC6XTOZLA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221316944 (1h51m51.002250328s) [seg_pos={0 0}, upload=1695]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:43:35.250332304 (1h51m51.031265679s) [upload=1695]: Done uploading segments... Jul 26 23:43:35.250361467 (1h51m51.031294841s) [upload=1695]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.03150034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032031617s FAIL (2025-07-26 23:43:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.032061499s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX......] duration: 9.545673361s FAIL (2025-07-26 23:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.545696426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.030371277s FAIL (2025-07-26 23:43:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 3XARCEPTK4JYBWH2EUKMYTM62MULCUUFWOTC5M3BHWK7A, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981083747 (1h51m9.762017143s) [noise=true , address=45.91.116.5:4297 , piece_id=25FDUAXC2C46CKAR4PJWS7CMVPOBWA6AQKMIZWJUBKUYF5RF67RQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981492105 (1h51m9.76242548s) [noise=true , address=104.36.82.10:4077 , piece_id=OFBYIX2D5OOYKFB6A7RJGF2GW4N7HSHUQAMJRMNLEW7QFLQ5AO5Q, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981516788 (1h51m9.762450154s) [noise=true , address=104.36.82.11:4115 , piece_id=GSJDZPDYFA7DMNTJ2TJFFUB4NCD6OCCZXH5YCICENLSB7RHPFD3Q, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981849577 (1h51m9.762782978s) [noise=true , address=77.90.31.19:4841 , piece_id=LDYGHNFSQFAFN4O3I5BUFO64NAG6XYFGVK462CBYV24TJ52HU2EA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981870660 (1h51m9.762804039s) [noise=true , address=104.36.82.8:4972 , piece_id=DY6BGVJ5LKR6UUAJJVTOHU7YFHG6H6EZAKAKONZE5TP6YMAVW2ZQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982170893 (1h51m9.763104268s) [noise=true , address=104.36.82.6:4873 , piece_id=5TKMXPN6YIVUNWM432J3SL7BQLVBJJ74UMQXC4WJHBIE65JS5PHA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982235052 (1h51m9.763168417s) [noise=true , address=104.36.82.13:4187 , piece_id=OWUCC5OVAOUEUXV2D43AOIJ6I7W2OPA6MJ6ENV4XPGGDITB3JCEQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982584542 (1h51m9.763517917s) [noise=true , address=45.91.116.6:4974 , piece_id=4UFEKAEIBJNASQIVZM36MFXWRBU3SRGZJ34FQTHK2HKZBNJZUERQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.983051143 (1h51m9.763984526s) [noise=true , address=104.36.82.14:4222 , piece_id=JA7RJIRPTADTDJ73U6KLAFBRUETJR4ADEZLJZZJHWSQJ23F4MMEA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.983636840 (1h51m9.764570218s) [noise=true , address=104.36.82.6:4875 , piece_id=SBWKH4VO7NPZTXUA24WEGKR37EFAVKAQIASWW63H4SBX2BZ6PKBA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984303355 (1h51m9.765236733s) [noise=true , address=104.36.82.12:4150 , piece_id=G25GIRL2W6DRKVSMO3NCQNQBNKB62BTCILHEBN4V47VCQRJ4F6FA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984627738 (1h51m9.765561119s) [noise=true , address=104.36.82.9:4006 , piece_id=A6I7YGR6CUHPDDSDTWCVQ5KL667SRBD7YMFVTZKYWVQST4S35XYA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984886950 (1h51m9.765820326s) [noise=true , address=104.36.82.7:4912 , piece_id=MVVOJM6FSRRPWCQO3JVRMZEOSKHGKMAYOGG3WUUAIRFPOX2LZRIQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.985246356 (1h51m9.766179732s) [noise=true , address=104.36.82.11:4113 , piece_id=JYVFDBOGWQOKH55W3YC5ZUD7PDBOGEEY4WUGAEQZ4OINYM6U2B6A, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.985640510 (1h51m9.766573886s) [noise=true , address=45.91.116.10:4081 , piece_id=6H4I445HNR3Y23Q4QD3WRHSPQ5775KSEMOC2WGVN3QSFO4XS4ADA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:54.046064822 (1h51m9.826998196s) [upload=1703]: Next returned nil segment Jul 26 23:42:54.046087221 (1h51m9.827020587s) [upload=1703]: Waiting for error group managing segments... Jul 26 23:42:54.049899328 (1h51m9.830832694s) [noise=true , address=45.91.116.10:4081 , piece_id=6H4I445HNR3Y23Q4QD3WRHSPQ5775KSEMOC2WGVN3QSFO4XS4ADA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.051776328 (1h51m9.832709706s) [noise=true , address=45.91.116.6:4974 , piece_id=4UFEKAEIBJNASQIVZM36MFXWRBU3SRGZJ34FQTHK2HKZBNJZUERQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.063612686 (1h51m9.844546048s) [noise=true , address=77.90.31.18:4805 , piece_id=ZKZAFDU3XARCEPTK4JYBWH2EUKMYTM62MULCUUFWOTC5M3BHWK7A, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.416235746 (1h51m10.197169129s) [noise=true , address=45.91.116.5:4297 , piece_id=25FDUAXC2C46CKAR4PJWS7CMVPOBWA6AQKMIZWJUBKUYF5RF67RQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:56.087754955 (1h51m11.868688322s) [noise=true , address=77.90.31.19:4841 , piece_id=LDYGHNFSQFAFN4O3I5BUFO64NAG6XYFGVK462CBYV24TJ52HU2EA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:43:23.916186840 (1h51m39.697120208s) [noise=true , address=104.36.82.11:4113 , piece_id=JYVFDBOGWQOKH55W3YC5ZUD7PDBOGEEY4WUGAEQZ4OINYM6U2B6A, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:45606->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916220069 (1h51m39.697153451s) [noise=true , address=104.36.82.9:4006 , piece_id=A6I7YGR6CUHPDDSDTWCVQ5KL667SRBD7YMFVTZKYWVQST4S35XYA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56338->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916255827 (1h51m39.697189201s) [noise=true , address=104.36.82.10:4077 , piece_id=OFBYIX2D5OOYKFB6A7RJGF2GW4N7HSHUQAMJRMNLEW7QFLQ5AO5Q, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916309256 (1h51m39.697242635s) [noise=true , address=104.36.82.13:4187 , piece_id=OWUCC5OVAOUEUXV2D43AOIJ6I7W2OPA6MJ6ENV4XPGGDITB3JCEQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:49964->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916333765 (1h51m39.697267132s) [noise=true , address=104.36.82.7:4912 , piece_id=MVVOJM6FSRRPWCQO3JVRMZEOSKHGKMAYOGG3WUUAIRFPOX2LZRIQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:58642->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916360143 (1h51m39.697293518s) [noise=true , address=104.36.82.11:4115 , piece_id=GSJDZPDYFA7DMNTJ2TJFFUB4NCD6OCCZXH5YCICENLSB7RHPFD3Q, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:39048->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916409312 (1h51m39.697342676s) [noise=true , address=104.36.82.8:4972 , piece_id=DY6BGVJ5LKR6UUAJJVTOHU7YFHG6H6EZAKAKONZE5TP6YMAVW2ZQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:41800->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916436941 (1h51m39.697370316s) [noise=true , address=104.36.82.14:4222 , piece_id=JA7RJIRPTADTDJ73U6KLAFBRUETJR4ADEZLJZZJHWSQJ23F4MMEA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:50074->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916455864 (1h51m39.697389229s) [noise=true , address=104.36.82.6:4873 , piece_id=5TKMXPN6YIVUNWM432J3SL7BQLVBJJ74UMQXC4WJHBIE65JS5PHA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:53064->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916498734 (1h51m39.69743211s) [noise=true , address=104.36.82.6:4875 , piece_id=SBWKH4VO7NPZTXUA24WEGKR37EFAVKAQIASWW63H4SBX2BZ6PKBA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:39380->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916552631 (1h51m39.697486011s) [noise=true , address=104.36.82.12:4150 , piece_id=G25GIRL2W6DRKVSMO3NCQNQBNKB62BTCILHEBN4V47VCQRJ4F6FA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:35634->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916592747 (1h51m39.697526129s) [seg_pos={0 0}, upload=1703]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:43:23.944310881 (1h51m39.725244256s) [upload=1703]: Done uploading segments... Jul 26 23:43:23.944334367 (1h51m39.725267741s) [upload=1703]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.030394212s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.....] duration: 1.887178471s FAIL (2025-07-26 23:42:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.887207055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.030356962s FAIL (2025-07-26 23:42:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== upload=1680]: Done uploading segments... Jul 26 23:42:06.214612402 (1h50m21.995545781s) [upload=1680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:42:06.305964647 (1h50m22.086898015s) [upload=1688]: Starting upload Jul 26 23:42:06.306063786 (1h50m22.086997165s) [upload=1688]: Done starting upload Jul 26 23:42:06.306143352 (1h50m22.087076719s) [upload=1688]: Uploading segments... Jul 26 23:42:06.306458206 (1h50m22.087391573s) [upload=1688]: Got next segment. Inline: false Jul 26 23:42:06.306520005 (1h50m22.087453388s) [upload=1688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:06.306526960 (1h50m22.087460338s) [upload=1688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:06.334335414 (1h50m22.115268803s) [seg_pos={0 0}, upload=1688]: Begin upload segment... Jul 26 23:42:06.334433210 (1h50m22.115366599s) [noise=true , address=104.36.82.3:4767 , piece_id=VQD4TZ7FGUJSZL3GM6BZMY5ZPR3VWLWSFUJSJT5SHGSYREXPTLNQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.334967292 (1h50m22.115900683s) [noise=true , address=104.36.82.10:4077 , piece_id=F4PNYXCLAWQKB3D4RR3MBWV6CMWVNAW6NAIZO4XFALQIIAFIWC2Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.335127934 (1h50m22.116061328s) [seg_pos={0 0}, upload=1688]: Done begin upload segment. Jul 26 23:42:06.335185809 (1h50m22.116119195s) [noise=true , address=77.90.31.16:4733 , piece_id=DJYNXBBGXFH4OZH3CRH4IP62TXMCLLGU5S5HZ3Q6U6LHHZSO6UZA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.335752603 (1h50m22.116685985s) [noise=true , address=45.91.116.4:4938 , piece_id=L24WSVKYFX3CXOSUWIVIIDTK5XMPMK6GN4BNJYRESO6OFZSWBZ5Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.336310404 (1h50m22.117243793s) [noise=true , address=104.36.82.16:4295 , piece_id=PHQLTJRH2LCFMBPWSZ4MIFSISBCRMGW6QBUVHKXDRLI7SGYYR32A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.336775211 (1h50m22.117708597s) [noise=true , address=77.90.31.13:4628 , piece_id=ANMDTRQ72JZXJZHRF4OSBLQR2XNHHH5X2SGSAMMN4A6A3G64Q3IA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337173662 (1h50m22.118107051s) [noise=true , address=45.91.116.13:4154 , piece_id=MD7PI4MKYZL5UWWTYGOM2KEFAVUUNX7JH7UH2WDCJX3MFSPRSA5Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337563479 (1h50m22.118496868s) [noise=true , address=45.91.116.6:4973 , piece_id=6BLIJOO6U25IXC5XIWFL3PX5Q3JBXIIYRS6HHLMFERYQ6MJI3G7A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337974126 (1h50m22.118907516s) [noise=true , address=104.36.82.7:4909 , piece_id=RU6VERJ57PKID62X7ANIZQWSECLR7UYOCZNJMUDYDYSI4HXWYOBQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.338395884 (1h50m22.119329275s) [noise=true , address=104.36.82.6:4874 , piece_id=7BQ5N5PORURDW332WGJFGTLRSWCJZTMTJFCCI2BNA6YET5DGS4AA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.338817490 (1h50m22.119750876s) [noise=true , address=45.91.116.14:4189 , piece_id=HNFVRFSSIJ6ZXTLRJSSY36GYA4FTR5575ROVESYPDZTWF5INAZ4A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.339265321 (1h50m22.120198707s) [noise=true , address=104.36.82.15:4257 , piece_id=SQSCUDWJVGXQHAGPPCGQC5CDQFOKJUNKTECFRL26ENN4RAEI2LFQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.340228672 (1h50m22.121162061s) [noise=true , address=77.90.31.19:4841 , piece_id=WO32ZKFOQJM3TP7AXVMUJ6RLAK6M3Z2XTMQSHCFZ7F4443OLBVHA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.340809116 (1h50m22.121742483s) [noise=true , address=45.91.116.5:4297 , piece_id=UGJNHDSFYMIZXY4AKNXQVC7PDVGUTUW5P4JSWVBGJ5FQSV2JM2MA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.341233170 (1h50m22.122166538s) [noise=true , address=104.36.82.13:4186 , piece_id=JHWUTMTLE6WNEDVE7YMPQTJKRLPJNQZL7C4DMAWJLUW736F7R6XQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.341624887 (1h50m22.122558274s) [noise=true , address=45.91.116.8:4045 , piece_id=KUXVDG2OV7MLEXIMRH2YSFHXZVJWQJPX3AWS7F52S4M3CUU2HEBQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.469607677 (1h50m22.250541057s) [upload=1688]: Next returned nil segment Jul 26 23:42:06.469640945 (1h50m22.250574324s) [upload=1688]: Waiting for error group managing segments... Jul 26 23:42:06.475366026 (1h50m22.256299392s) [noise=true , address=45.91.116.6:4973 , piece_id=6BLIJOO6U25IXC5XIWFL3PX5Q3JBXIIYRS6HHLMFERYQ6MJI3G7A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.475737541 (1h50m22.25667091s) [noise=true , address=45.91.116.13:4154 , piece_id=MD7PI4MKYZL5UWWTYGOM2KEFAVUUNX7JH7UH2WDCJX3MFSPRSA5Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.476295326 (1h50m22.257228701s) [noise=true , address=45.91.116.8:4045 , piece_id=KUXVDG2OV7MLEXIMRH2YSFHXZVJWQJPX3AWS7F52S4M3CUU2HEBQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.476502798 (1h50m22.257436168s) [noise=true , address=45.91.116.4:4938 , piece_id=L24WSVKYFX3CXOSUWIVIIDTK5XMPMK6GN4BNJYRESO6OFZSWBZ5Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.513349488 (1h50m22.294282857s) [noise=true , address=45.91.116.5:4297 , piece_id=UGJNHDSFYMIZXY4AKNXQVC7PDVGUTUW5P4JSWVBGJ5FQSV2JM2MA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.574425571 (1h50m22.355358943s) [noise=true , address=45.91.116.14:4189 , piece_id=HNFVRFSSIJ6ZXTLRJSSY36GYA4FTR5575ROVESYPDZTWF5INAZ4A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:07.860144723 (1h50m23.641078102s) [noise=true , address=77.90.31.19:4841 , piece_id=WO32ZKFOQJM3TP7AXVMUJ6RLAK6M3Z2XTMQSHCFZ7F4443OLBVHA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:07.926074677 (1h50m23.707008044s) [noise=true , address=77.90.31.13:4628 , piece_id=ANMDTRQ72JZXJZHRF4OSBLQR2XNHHH5X2SGSAMMN4A6A3G64Q3IA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:08.680110898 (1h50m24.461044262s) [noise=true , address=77.90.31.16:4733 , piece_id=DJYNXBBGXFH4OZH3CRH4IP62TXMCLLGU5S5HZ3Q6U6LHHZSO6UZA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:36.271778029 (1h50m52.05271141s) [noise=true , address=104.36.82.13:4186 , piece_id=JHWUTMTLE6WNEDVE7YMPQTJKRLPJNQZL7C4DMAWJLUW736F7R6XQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:52146->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271862783 (1h50m52.052796166s) [noise=true , address=104.36.82.7:4909 , piece_id=RU6VERJ57PKID62X7ANIZQWSECLR7UYOCZNJMUDYDYSI4HXWYOBQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271924435 (1h50m52.052857814s) [noise=true , address=104.36.82.10:4077 , piece_id=F4PNYXCLAWQKB3D4RR3MBWV6CMWVNAW6NAIZO4XFALQIIAFIWC2Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:53766->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271998960 (1h50m52.052932343s) [noise=true , address=104.36.82.6:4874 , piece_id=7BQ5N5PORURDW332WGJFGTLRSWCJZTMTJFCCI2BNA6YET5DGS4AA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:52030->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272011472 (1h50m52.052944845s) [noise=true , address=104.36.82.16:4295 , piece_id=PHQLTJRH2LCFMBPWSZ4MIFSISBCRMGW6QBUVHKXDRLI7SGYYR32A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:53500->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272024315 (1h50m52.052957688s) [noise=true , address=104.36.82.3:4767 , piece_id=VQD4TZ7FGUJSZL3GM6BZMY5ZPR3VWLWSFUJSJT5SHGSYREXPTLNQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272255843 (1h50m52.053189221s) [noise=true , address=104.36.82.15:4257 , piece_id=SQSCUDWJVGXQHAGPPCGQC5CDQFOKJUNKTECFRL26ENN4RAEI2LFQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:46470->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272285115 (1h50m52.053218506s) [seg_pos={0 0}, upload=1688]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:42:36.298684003 (1h50m52.079617373s) [upload=1688]: Done uploading segments... Jul 26 23:42:36.298733221 (1h50m52.07966661s) [upload=1688]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030377305s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.....XX...X...X..XX.X...X..X.......X...X....XX..............] duration: 42.580725782s FAIL (2025-07-26 23:42:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.580749279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033203436s FAIL (2025-07-26 23:42:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 40.038765653s) [upload=1676]: Got next segment. Inline: false Jul 26 23:41:24.257891756 (1h49m40.038825132s) [upload=1676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:41:24.257905676 (1h49m40.038839039s) [upload=1676]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:41:24.368560933 (1h49m40.14949431s) [seg_pos={0 0}, upload=1676]: Begin upload segment... Jul 26 23:41:24.368634748 (1h49m40.149568125s) [seg_pos={0 0}, upload=1676]: Done begin upload segment. Jul 26 23:41:24.368661312 (1h49m40.149594696s) [noise=true , address=64.49.13.139:4478 , piece_id=SEALYRUU6WMN5DFB5HTFJQ4KGR4MOMGM4RNXEBEH4SRU55YXQA6Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.368671444 (1h49m40.149604823s) [noise=true , address=104.36.82.46:4502 , piece_id=GIYTZMLOJ3OUGDWGH3TCKTHBPTGSZTACX7EB3SNPHHG3TDWGNLOQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.368758256 (1h49m40.149691641s) [noise=true , address=45.91.116.9:4273 , piece_id=SQEXEHC5KJH6T6METISM65F7UM2ZMNGIRPSS5HDU2BV7UTAGA6AQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369206778 (1h49m40.150140163s) [noise=true , address=79.127.205.234:4616 , piece_id=RQXLOBMU7RVBZ2IU3SGXNI65FNNHHISIER6X2ATLMFTTNHFAOULQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369671630 (1h49m40.150605015s) [noise=true , address=64.49.13.137:4502 , piece_id=7FS5TCVFPX57CCC4G4HIDA2ANR5KZT7ZDBMUAVAYEAZ5RNPYE5JA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369996867 (1h49m40.150930246s) [noise=true , address=77.90.31.12:4608 , piece_id=L462NKLDIOXJS4APFQBLL3LD6XMOVMN7COOPRJPSRPYM622BT65Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.370114633 (1h49m40.151048018s) [noise=true , address=104.36.82.53:4129 , piece_id=7R77RGD2BNDAT3GWPOGJ3DUO74H7BPR7EF7HDSCLPQBSKNR5BBIQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.370583399 (1h49m40.151516783s) [noise=true , address=79.127.220.98:4422 , piece_id=R473XJFLXHJYFDAMFOLY3DGANAJPA2NGWQWQDGV3G6QIO6U2PQYA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.371130862 (1h49m40.152064285s) [noise=true , address=79.127.205.246:4273 , piece_id=HSXMYGCUUELLUGLCJU4YMF2M5KDZ6AJ2IOCP2LTD7KSCC2ZUB4BQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.371661566 (1h49m40.152594951s) [noise=true , address=104.36.82.20:4471 , piece_id=334D667P6G4477HV737Y632Y45JC5DBFJ4UMJ2FPS4GMKOLGWDLQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.372506630 (1h49m40.153440015s) [noise=true , address=156.146.43.225:4448 , piece_id=Y7NN643J5DGFQBRXUYU6Z3I5STAK7EMKUZQOYRNKPLW425W4XSTA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.373070034 (1h49m40.154003412s) [noise=true , address=79.127.201.215:4348 , piece_id=KWO7TJFJCNXQ7FJZNH6WNVUT4YTYINRQPBFJOQB26WLPXBRDUUSA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.374911360 (1h49m40.155844737s) [noise=true , address=77.90.31.18:4809 , piece_id=ATDNAIDAY7P2S4V6L32DLTMKYV2NEO34Z7RT4VXEA5LDAE62JBUA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.376159240 (1h49m40.15709263s) [noise=true , address=45.91.116.14:4222 , piece_id=KLJG4TGSOIZFZH5IRCZW6SPAES4CKP77UQFX54HZJWHIPQIP65JA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.376631014 (1h49m40.157564399s) [noise=true , address=104.36.82.4:4819 , piece_id=EQMTLTI6TZ5RTOIQWT7TZAWOVLXEXXCTKZ7J6GXOXOCD3JWLTE6Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377066117 (1h49m40.157999505s) [noise=true , address=79.127.205.233:4645 , piece_id=TDCWGLKTOEBVK3FSFEVSDOQTYOWD64SMMGGRLZV3TQA6UUODSTUA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377503897 (1h49m40.158437279s) [noise=true , address=104.36.82.55:4207 , piece_id=2OAPI5VWAYZOVWJBSLUAZW5YD3ABDJYVCNHHREWP3BWM375PGYCA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377954027 (1h49m40.158887411s) [noise=true , address=104.36.82.47:4535 , piece_id=CX42IE2RSAXADFH4MXY5NFMNX7WO5BELSSVZRVALINV37M37O3XA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.453937149 (1h49m40.234870521s) [upload=1676]: Next returned nil segment Jul 26 23:41:24.453959086 (1h49m40.234892465s) [upload=1676]: Waiting for error group managing segments... Jul 26 23:41:24.458611613 (1h49m40.23954499s) [noise=true , address=77.90.31.18:4809 , piece_id=ATDNAIDAY7P2S4V6L32DLTMKYV2NEO34Z7RT4VXEA5LDAE62JBUA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.476428918 (1h49m40.257362296s) [noise=true , address=77.90.31.12:4608 , piece_id=L462NKLDIOXJS4APFQBLL3LD6XMOVMN7COOPRJPSRPYM622BT65Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.951783253 (1h49m40.732716618s) [noise=true , address=79.127.205.233:4645 , piece_id=TDCWGLKTOEBVK3FSFEVSDOQTYOWD64SMMGGRLZV3TQA6UUODSTUA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.958568263 (1h49m40.739501634s) [noise=true , address=79.127.205.246:4273 , piece_id=HSXMYGCUUELLUGLCJU4YMF2M5KDZ6AJ2IOCP2LTD7KSCC2ZUB4BQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.976208837 (1h49m40.757142209s) [noise=true , address=79.127.205.234:4616 , piece_id=RQXLOBMU7RVBZ2IU3SGXNI65FNNHHISIER6X2ATLMFTTNHFAOULQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:25.045268499 (1h49m40.826201862s) [noise=true , address=64.49.13.139:4478 , piece_id=SEALYRUU6WMN5DFB5HTFJQ4KGR4MOMGM4RNXEBEH4SRU55YXQA6Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:25.079236000 (1h49m40.860169366s) [noise=true , address=64.49.13.137:4502 , piece_id=7FS5TCVFPX57CCC4G4HIDA2ANR5KZT7ZDBMUAVAYEAZ5RNPYE5JA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:26.565677607 (1h49m42.346610973s) [noise=true , address=156.146.43.225:4448 , piece_id=Y7NN643J5DGFQBRXUYU6Z3I5STAK7EMKUZQOYRNKPLW425W4XSTA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:28.689509351 (1h49m44.47044272s) [noise=true , address=79.127.220.98:4422 , piece_id=R473XJFLXHJYFDAMFOLY3DGANAJPA2NGWQWQDGV3G6QIO6U2PQYA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:31.511898241 (1h49m47.292831605s) [noise=true , address=45.91.116.14:4222 , piece_id=KLJG4TGSOIZFZH5IRCZW6SPAES4CKP77UQFX54HZJWHIPQIP65JA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:35.832008124 (1h49m51.612941487s) [noise=true , address=45.91.116.9:4273 , piece_id=SQEXEHC5KJH6T6METISM65F7UM2ZMNGIRPSS5HDU2BV7UTAGA6AQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:49.897102300 (1h50m5.678035672s) [noise=true , address=79.127.201.215:4348 , piece_id=KWO7TJFJCNXQ7FJZNH6WNVUT4YTYINRQPBFJOQB26WLPXBRDUUSA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:42:24.227528079 (1h50m40.008461444s) [noise=true , address=104.36.82.47:4535 , piece_id=CX42IE2RSAXADFH4MXY5NFMNX7WO5BELSSVZRVALINV37M37O3XA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227587774 (1h50m40.008521138s) [noise=true , address=104.36.82.53:4129 , piece_id=7R77RGD2BNDAT3GWPOGJ3DUO74H7BPR7EF7HDSCLPQBSKNR5BBIQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227712284 (1h50m40.00864566s) [noise=true , address=104.36.82.20:4471 , piece_id=334D667P6G4477HV737Y632Y45JC5DBFJ4UMJ2FPS4GMKOLGWDLQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12DMRnzj9hpP9tRn5DUz9WcU3N4jEZ3CZ9saVTFa1tefjDNs72r, address:104.36.82.20:4471): protocol: rpc: dial tcp 104.36.82.20:4471: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227760043 (1h50m40.008693409s) [noise=true , address=104.36.82.46:4502 , piece_id=GIYTZMLOJ3OUGDWGH3TCKTHBPTGSZTACX7EB3SNPHHG3TDWGNLOQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227779152 (1h50m40.008712527s) [noise=true , address=104.36.82.4:4819 , piece_id=EQMTLTI6TZ5RTOIQWT7TZAWOVLXEXXCTKZ7J6GXOXOCD3JWLTE6Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12mh9u38YmD7NcPA5uB1KLaw8dToTUKywJsi1MhhSc17LXo15Gb, address:104.36.82.4:4819): protocol: rpc: dial tcp 104.36.82.4:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227821940 (1h50m40.008755317s) [noise=true , address=104.36.82.55:4207 , piece_id=2OAPI5VWAYZOVWJBSLUAZW5YD3ABDJYVCNHHREWP3BWM375PGYCA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:36586->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227859111 (1h50m40.008792492s) [seg_pos={0 0}, upload=1676]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:42:24.258470670 (1h50m40.039404039s) [upload=1676]: Done uploading segments... Jul 26 23:42:24.258497044 (1h50m40.03943041s) [upload=1676]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.033227379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [.........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025847775s FAIL (2025-07-26 23:42:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025888515s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030066823s FAIL (2025-07-26 23:42:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== TAGMQJGUC7HAHAJUI7I5ITNSEFAAUIK653A, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.255757677 (1h49m52.036691061s) [noise=true , address=104.36.82.14:4221 , piece_id=BAZUCKNRWY6FIMXJXC5SDMZ3NNMGQLKJYVG2ZNXWMXUIAXZD4UKQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.256412961 (1h49m52.03734635s) [noise=true , address=104.36.82.2:4731 , piece_id=FNTV3CVI67HG4BGPO4CGMY2XB5AFFCA5JN7OKYA5TX4D6TSEIKHA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.256999306 (1h49m52.037932684s) [noise=true , address=104.36.82.12:4149 , piece_id=EPG5G6CKA3REHAH73ONGOIDNGFVYXRHBBWCEBB7WL5M66IBPC2KQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.257571391 (1h49m52.038504777s) [noise=true , address=45.91.116.6:4975 , piece_id=WXCTWLZWW23ZIDSORXNWNO7AHTRUWGEDUSY7X66BUHR6ZQSDJTBA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.258077324 (1h49m52.03901071s) [noise=true , address=104.36.82.13:4185 , piece_id=YUNDQD3US3UDJ2YBTSHFQJN3WGSNJYE6BLHYYK5RJ653M27JUZOA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.258637468 (1h49m52.039570855s) [noise=true , address=77.90.31.14:4661 , piece_id=RQ5BJWUNECNOOTVGJ257N2KHI5FTZ4UQ7RP7HTXLCYNLN64UXT2Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.259245941 (1h49m52.040179331s) [noise=true , address=104.36.82.5:4840 , piece_id=TAWV7WNIQVSLAIKJLDMP2RIPDN3J5ARSKBFTZVKPRIUS4HS4BQ4Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.259673432 (1h49m52.040606817s) [noise=true , address=104.36.82.7:4910 , piece_id=ZD7PQE4NP7QMKW2DA2UF47BDYZNPKDF5AO7SSDUOEDD7U2GYIISQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.260261110 (1h49m52.041194499s) [noise=true , address=104.36.82.11:4113 , piece_id=KMSGRHSQCYB77VCEQ7LXL4XSAWEJN6ZXUVINQHWCWSDWQVVY64YQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.260716032 (1h49m52.041649416s) [noise=true , address=77.90.31.10:4517 , piece_id=N6FQ6QET5Y6KO7KA7AUDDBKZVRG7UNCQVYDBR4TQP3YHSOO74E6Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.262834753 (1h49m52.04376814s) [noise=true , address=77.90.31.16:4733 , piece_id=2J75KCPYIOOC4WM6WPHH4HS2VJMK6MK6ICTUTACP4YLCRXATZBFQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.263419790 (1h49m52.044353176s) [noise=true , address=104.36.82.10:4078 , piece_id=V5OP6J7B5K3UFIWDCKWXMTHIB7SG35MI7W75RHWJDYE25N6FZELQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.263929602 (1h49m52.044862988s) [noise=true , address=104.36.82.8:4971 , piece_id=PLZKBWZ2S3VCGZFDUYB6JJZRACJY4BSAKH65VPJ4A3TGLRZK7JFQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.264484642 (1h49m52.04541803s) [noise=true , address=77.90.31.12:4589 , piece_id=3BSILG2WOW7FGFQAQU64WOIDNBZ7Z6DTLFMRMYT6HATBAZPR4IXA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.264974774 (1h49m52.04590816s) [noise=true , address=104.36.82.14:4223 , piece_id=J5IUFPDIZNMPDRDDXU25DNMHZVAXL6XSHYM5AU2WEWF24EYPZYNQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.351658895 (1h49m52.132592281s) [upload=1680]: Next returned nil segment Jul 26 23:41:36.351682811 (1h49m52.132616199s) [upload=1680]: Waiting for error group managing segments... Jul 26 23:41:36.356875043 (1h49m52.137808408s) [noise=true , address=77.90.31.12:4589 , piece_id=3BSILG2WOW7FGFQAQU64WOIDNBZ7Z6DTLFMRMYT6HATBAZPR4IXA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.357656780 (1h49m52.138590144s) [noise=true , address=77.90.31.14:4661 , piece_id=RQ5BJWUNECNOOTVGJ257N2KHI5FTZ4UQ7RP7HTXLCYNLN64UXT2Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.357722689 (1h49m52.13865605s) [noise=true , address=45.91.116.6:4975 , piece_id=WXCTWLZWW23ZIDSORXNWNO7AHTRUWGEDUSY7X66BUHR6ZQSDJTBA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.821001688 (1h49m52.601935057s) [noise=true , address=45.91.116.5:4298 , piece_id=OEMTYFGTREMSORNMTTAGMQJGUC7HAHAJUI7I5ITNSEFAAUIK653A, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:38.345059193 (1h49m54.125992561s) [noise=true , address=77.90.31.16:4733 , piece_id=2J75KCPYIOOC4WM6WPHH4HS2VJMK6MK6ICTUTACP4YLCRXATZBFQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:42:06.187137843 (1h50m21.968071231s) [noise=true , address=104.36.82.13:4185 , piece_id=YUNDQD3US3UDJ2YBTSHFQJN3WGSNJYE6BLHYYK5RJ653M27JUZOA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187229901 (1h50m21.968163281s) [noise=true , address=104.36.82.2:4731 , piece_id=FNTV3CVI67HG4BGPO4CGMY2XB5AFFCA5JN7OKYA5TX4D6TSEIKHA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:60718->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187420240 (1h50m21.968353627s) [noise=true , address=104.36.82.11:4113 , piece_id=KMSGRHSQCYB77VCEQ7LXL4XSAWEJN6ZXUVINQHWCWSDWQVVY64YQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:38356->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187508196 (1h50m21.968441577s) [noise=true , address=104.36.82.10:4078 , piece_id=V5OP6J7B5K3UFIWDCKWXMTHIB7SG35MI7W75RHWJDYE25N6FZELQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56256->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187667620 (1h50m21.968601s) [noise=true , address=77.90.31.10:4517 , piece_id=N6FQ6QET5Y6KO7KA7AUDDBKZVRG7UNCQVYDBR4TQP3YHSOO74E6Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:56852->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187734781 (1h50m21.96866815s) [noise=true , address=104.36.82.8:4971 , piece_id=PLZKBWZ2S3VCGZFDUYB6JJZRACJY4BSAKH65VPJ4A3TGLRZK7JFQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:59706->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187798359 (1h50m21.968731739s) [noise=true , address=104.36.82.12:4149 , piece_id=EPG5G6CKA3REHAH73ONGOIDNGFVYXRHBBWCEBB7WL5M66IBPC2KQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:45102->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187939106 (1h50m21.968872485s) [noise=true , address=104.36.82.7:4910 , piece_id=ZD7PQE4NP7QMKW2DA2UF47BDYZNPKDF5AO7SSDUOEDD7U2GYIISQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:34202->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188017733 (1h50m21.968951116s) [noise=true , address=104.36.82.14:4223 , piece_id=J5IUFPDIZNMPDRDDXU25DNMHZVAXL6XSHYM5AU2WEWF24EYPZYNQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:52750->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188220167 (1h50m21.96915355s) [noise=true , address=104.36.82.14:4221 , piece_id=BAZUCKNRWY6FIMXJXC5SDMZ3NNMGQLKJYVG2ZNXWMXUIAXZD4UKQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:36226->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188241038 (1h50m21.969174418s) [noise=true , address=104.36.82.5:4840 , piece_id=TAWV7WNIQVSLAIKJLDMP2RIPDN3J5ARSKBFTZVKPRIUS4HS4BQ4Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:50076->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188300179 (1h50m21.969233568s) [seg_pos={0 0}, upload=1680]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:42:06.214580561 (1h50m21.995513937s) [upload=1680]: Done uploading segments... Jul 26 23:42:06.214612402 (1h50m21.995545781s) [upload=1680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.03010557s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX....] duration: 20.276322076s FAIL (2025-07-26 23:41:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.276347987s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......XX...X...X..XX.X...X..X.......X...X....XX.............] duration: 28.194715909s FAIL (2025-07-26 23:41:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.194772421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035685682s FAIL (2025-07-26 23:41:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 48m40.051376791s) [noise=true , address=77.90.31.10:4530 , piece_id=FJDP7PPG6HBIEE3TMZVCKSXMHBHHXWL3PQEFDSULKPYUZBXLRPYA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270467697 (1h48m40.051401084s) [noise=true , address=79.127.205.233:4624 , piece_id=BLM6NYO6D4OAYE72DT73MV2BEL3QKMNHHPZDETULH7UJIQ53GIPA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270917792 (1h48m40.051851171s) [noise=true , address=79.127.205.235:4688 , piece_id=B7N5D6L2TIHJE2NCC3T4EW7V6AF2OSOG7JP7FZVOW5KLIB3VHYQA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270998428 (1h48m40.051931804s) [noise=true , address=64.49.13.140:4508 , piece_id=4ZVYHRUAFN5SCRH66KURZVVOCA4TDFQATKM6RPTYMYM7NPIVE35A, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271350958 (1h48m40.052284346s) [noise=true , address=104.36.82.46:4515 , piece_id=R2BJK7TSBUPWOS6R5MZ772RYT5HUFZHMW2Q3MUWAOWXZ5BEZUPTA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271473112 (1h48m40.052406516s) [noise=true , address=104.36.82.13:4201 , piece_id=IZNCNXAWBMETWVO5CVPXRJD55OPS325LAQPMMST6JO7Z6WSNE4JQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271698545 (1h48m40.052631923s) [noise=true , address=104.36.82.54:4157 , piece_id=PWD7RXN2ZLEMM4G5EDQTQSDGZ6ZR2522JFOLEUUBVRZTUY4OMQ5Q, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272011165 (1h48m40.05294454s) [noise=true , address=79.127.205.245:4250 , piece_id=TTAME4MD4D4KDPRZVLK6TY6R5VT257IMQXJTWXQI3NQBCWPXAWHA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272019288 (1h48m40.052952671s) [noise=true , address=45.91.116.12:4230 , piece_id=5NJH6XVJXRLRVTXWLTL5RC6K2TPXJDYFFXYPXZ7BNCZ4YZE7Z3TQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272588816 (1h48m40.053522197s) [noise=true , address=79.127.205.234:4610 , piece_id=ZBW5XKDXFIVS6Y6ZTSSJTC4IFWQCNCOM7F2OAHD67K5NZCTDEFAA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272612677 (1h48m40.053546052s) [noise=true , address=77.90.31.13:4658 , piece_id=H2HTVHF57ES6KUO5AAZLFBUIBG67K6HTHWGYSHFMR5WYK3F5FMJQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272882654 (1h48m40.053816031s) [noise=true , address=64.49.13.138:4483 , piece_id=IUDDVA7UMZMQ5ZQAQEPPPWL4ULLMCJAPQX6U2UPP5GZ6AOF77OWQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273065664 (1h48m40.053999051s) [noise=true , address=104.36.82.47:4531 , piece_id=OM7UDG2S5GBKWPJCONEX6EGBEZMMFPXXROZQXUZ7S2GCC2MB4VWA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273228212 (1h48m40.054161593s) [noise=true , address=45.91.116.14:4194 , piece_id=KAGPM52MUDUX2J5W5OYJJQSTBMVDOZIJ7TZZUA5K3VWUMYWGLCDA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273523109 (1h48m40.054456489s) [noise=true , address=77.90.31.18:4809 , piece_id=GWOJI77VCVRVOEJS37LPVAF23P6UZVW6PRIJ4IXW2ZUGUHNDCKFQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273907133 (1h48m40.054840517s) [noise=true , address=104.36.82.56:4260 , piece_id=BEN7QYOJ5CPBFCQ4H5SSKKU67HNCZIFTDBWWLP2OHIBRUPILRWOQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.274559142 (1h48m40.055492521s) [noise=true , address=104.36.82.52:4090 , piece_id=AWX7GLYR6CSE4JFUM5F2JSGQLT7VJZGRQNMO7QBSF2EDMI5X5PSQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.274953607 (1h48m40.055886985s) [noise=true , address=79.127.219.38:4003 , piece_id=Y2ZCMRSFK476XM5FLL5INNFFPFDUJALFJJLUWQGWOOIVE7ROSQKQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.350887700 (1h48m40.131821067s) [upload=1663]: Next returned nil segment Jul 26 23:40:24.350903636 (1h48m40.131837011s) [upload=1663]: Waiting for error group managing segments... Jul 26 23:40:24.388668963 (1h48m40.16960233s) [noise=true , address=77.90.31.18:4809 , piece_id=GWOJI77VCVRVOEJS37LPVAF23P6UZVW6PRIJ4IXW2ZUGUHNDCKFQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.756826310 (1h48m40.537759674s) [noise=true , address=79.127.219.38:4003 , piece_id=Y2ZCMRSFK476XM5FLL5INNFFPFDUJALFJJLUWQGWOOIVE7ROSQKQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.840401438 (1h48m40.621334821s) [noise=true , address=79.127.205.245:4250 , piece_id=TTAME4MD4D4KDPRZVLK6TY6R5VT257IMQXJTWXQI3NQBCWPXAWHA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.863377794 (1h48m40.644311163s) [noise=true , address=79.127.205.235:4688 , piece_id=B7N5D6L2TIHJE2NCC3T4EW7V6AF2OSOG7JP7FZVOW5KLIB3VHYQA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:25.073974314 (1h48m40.854907696s) [noise=true , address=79.127.205.233:4624 , piece_id=BLM6NYO6D4OAYE72DT73MV2BEL3QKMNHHPZDETULH7UJIQ53GIPA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:25.159720016 (1h48m40.940653385s) [noise=true , address=79.127.205.234:4610 , piece_id=ZBW5XKDXFIVS6Y6ZTSSJTC4IFWQCNCOM7F2OAHD67K5NZCTDEFAA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:26.227989914 (1h48m42.00892328s) [noise=true , address=64.49.13.140:4508 , piece_id=4ZVYHRUAFN5SCRH66KURZVVOCA4TDFQATKM6RPTYMYM7NPIVE35A, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:28.218665173 (1h48m43.999598541s) [noise=true , address=77.90.31.13:4658 , piece_id=H2HTVHF57ES6KUO5AAZLFBUIBG67K6HTHWGYSHFMR5WYK3F5FMJQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:29.666636360 (1h48m45.447569751s) [noise=true , address=45.91.116.14:4194 , piece_id=KAGPM52MUDUX2J5W5OYJJQSTBMVDOZIJ7TZZUA5K3VWUMYWGLCDA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:30.524754720 (1h48m46.305688103s) [noise=true , address=45.91.116.12:4230 , piece_id=5NJH6XVJXRLRVTXWLTL5RC6K2TPXJDYFFXYPXZ7BNCZ4YZE7Z3TQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:30.866554622 (1h48m46.647488005s) [noise=true , address=64.49.13.138:4483 , piece_id=IUDDVA7UMZMQ5ZQAQEPPPWL4ULLMCJAPQX6U2UPP5GZ6AOF77OWQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:41:24.111643473 (1h49m39.892576852s) [noise=true , address=104.36.82.13:4201 , piece_id=IZNCNXAWBMETWVO5CVPXRJD55OPS325LAQPMMST6JO7Z6WSNE4JQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: write tcp 159.223.143.39:34040->104.36.82.13:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111808870 (1h49m39.892742251s) [noise=true , address=104.36.82.47:4531 , piece_id=OM7UDG2S5GBKWPJCONEX6EGBEZMMFPXXROZQXUZ7S2GCC2MB4VWA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111893200 (1h49m39.892826576s) [noise=true , address=104.36.82.56:4260 , piece_id=BEN7QYOJ5CPBFCQ4H5SSKKU67HNCZIFTDBWWLP2OHIBRUPILRWOQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111949764 (1h49m39.89288314s) [noise=true , address=104.36.82.52:4090 , piece_id=AWX7GLYR6CSE4JFUM5F2JSGQLT7VJZGRQNMO7QBSF2EDMI5X5PSQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112026515 (1h49m39.89295988s) [noise=true , address=77.90.31.10:4530 , piece_id=FJDP7PPG6HBIEE3TMZVCKSXMHBHHXWL3PQEFDSULKPYUZBXLRPYA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:41:24.112235751 (1h49m39.893169194s) [noise=true , address=104.36.82.54:4157 , piece_id=PWD7RXN2ZLEMM4G5EDQTQSDGZ6ZR2522JFOLEUUBVRZTUY4OMQ5Q, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112307985 (1h49m39.893241368s) [noise=true , address=104.36.82.46:4515 , piece_id=R2BJK7TSBUPWOS6R5MZ772RYT5HUFZHMW2Q3MUWAOWXZ5BEZUPTA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:46716->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112360351 (1h49m39.893293735s) [seg_pos={0 0}, upload=1663]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:41:24.145267611 (1h49m39.926200973s) [upload=1663]: Done uploading segments... Jul 26 23:41:24.145290823 (1h49m39.926224184s) [upload=1663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.035706309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [..........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026249983s FAIL (2025-07-26 23:41:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026273243s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......XX...X...X..XX.X...X..X.......X...X....XX............] duration: 43.132907093s FAIL (2025-07-26 23:41:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.132947676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...] duration: 9.113012538s FAIL (2025-07-26 23:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.113040771s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..] duration: 4.292889632s FAIL (2025-07-26 23:40:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.293007942s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.] duration: 28.068937431s FAIL (2025-07-26 23:40:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.068979967s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........XX...X...X..XX.X...X..X.......X...X....XX...........] duration: 40.224662823s FAIL (2025-07-26 23:40:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.224677536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1.76924134s FAIL (2025-07-26 23:40:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== : Starting upload Jul 26 23:39:53.651659456 (1h48m9.432592821s) [upload=1656]: Done starting upload Jul 26 23:39:53.651795715 (1h48m9.432729094s) [upload=1656]: Uploading segments... Jul 26 23:39:53.652022529 (1h48m9.432955897s) [upload=1656]: Got next segment. Inline: false Jul 26 23:39:53.652177070 (1h48m9.433110436s) [upload=1656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:53.652184134 (1h48m9.433117509s) [upload=1656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:53.760344617 (1h48m9.541278013s) [seg_pos={0 0}, upload=1656]: Begin upload segment... Jul 26 23:39:53.760424673 (1h48m9.541358039s) [seg_pos={0 0}, upload=1656]: Done begin upload segment. Jul 26 23:39:53.760459139 (1h48m9.541392526s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.760511419 (1h48m9.5414448s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761043421 (1h48m9.541976797s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761155851 (1h48m9.542089232s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761542565 (1h48m9.542475946s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761742109 (1h48m9.542675495s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762101713 (1h48m9.543035094s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762304875 (1h48m9.543238262s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762586015 (1h48m9.543519392s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762908490 (1h48m9.543841879s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763217573 (1h48m9.544150957s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763475029 (1h48m9.544408418s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763719412 (1h48m9.544652792s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763976761 (1h48m9.544910146s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764165834 (1h48m9.54509922s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764445958 (1h48m9.545379346s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.857635650 (1h48m9.638569016s) [upload=1656]: Next returned nil segment Jul 26 23:39:53.857653813 (1h48m9.638587183s) [upload=1656]: Waiting for error group managing segments... Jul 26 23:39:53.861881868 (1h48m9.642815249s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.861935675 (1h48m9.642869038s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862515297 (1h48m9.643448675s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862570367 (1h48m9.643503736s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.863057776 (1h48m9.643991146s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.866234319 (1h48m9.6471677s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.985553629 (1h48m9.766486998s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:40:23.614501940 (1h48m39.395435321s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:46296->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614816596 (1h48m39.395749978s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614954894 (1h48m39.395888279s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615058497 (1h48m39.39599188s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615118572 (1h48m39.396051953s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615198214 (1h48m39.396131594s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615286224 (1h48m39.3962196s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615304725 (1h48m39.396238092s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615342159 (1h48m39.396275535s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615382082 (1h48m39.396315457s) [seg_pos={0 0}, upload=1656]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:40:23.662236247 (1h48m39.443169685s) [upload=1656]: Done uploading segments... Jul 26 23:40:23.662284995 (1h48m39.443218371s) [upload=1656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 1.769266308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [X.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028528971s FAIL (2025-07-26 23:40:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Jul 26 23:39:24.055606865 (1h47m39.836540231s) [seg_pos={0 0}, upload=1648]: Done begin upload segment. Jul 26 23:39:24.055640127 (1h47m39.836573502s) [noise=true , address=104.36.82.55:4207 , piece_id=YVFUZ2LA4OA3AFJGEY7X3ZVRLZZXM3WNFSFVBOQ4BT4TJOF2OQEA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.055660702 (1h47m39.836594087s) [noise=true , address=104.36.82.46:4492 , piece_id=JMGB57YRCYCU6ADCVVDG5ELOEZMMRTB3BF7Q3ODKOHHDAH53BK4A, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056101217 (1h47m39.837034659s) [noise=true , address=104.36.82.52:4092 , piece_id=SM24GEERL7EHLUPKZ422HP75G3U5DGV76KBO27K2TZJESKA7JOKA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056216367 (1h47m39.837149744s) [noise=true , address=79.127.201.215:4353 , piece_id=2BMTPA7JVRDZCZUXEN46YW2GV4KJP4VUTXXWCKSADVFBS43V75PA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056443546 (1h47m39.837376985s) [noise=true , address=104.36.82.22:4537 , piece_id=7STK4LSTI6IWOWREQNSKQVH5ZGSQU3IBCJJFQQTPITK6HKB4GMGQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056785677 (1h47m39.837719055s) [noise=true , address=79.127.205.235:4691 , piece_id=DGWB6WXMIOYMYYD6S3KH52UZMQVXNUQB5CGODP47O4IM77MJHYUQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.059670543 (1h47m39.840603927s) [noise=true , address=77.90.31.12:4614 , piece_id=SJ5V3WFJVN3376OE4AIOLRUUWTAGJ3IHZM2WHBQ6D5PMLFMSA4LA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.060225984 (1h47m39.84115937s) [noise=true , address=79.127.205.234:4618 , piece_id=Z2QWR75CR7PO6LNLQKV62H4WQGAI26BKLTZN3GSXE2PT5XLHG6LA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.060778266 (1h47m39.841711651s) [noise=true , address=45.91.116.11:4121 , piece_id=QHOLWAUP4JJ2P4FSHOPIXQJFR2J67LZIKJJM4W7DV6WJG6JMLDIQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.061271482 (1h47m39.842204864s) [noise=true , address=79.127.205.233:4640 , piece_id=22M2A6452J6XA3JFCE3C6GGYNXHDLT7EYTBDIPICAGTY2DDSKFOA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.061704848 (1h47m39.842638234s) [noise=true , address=77.90.31.21:4912 , piece_id=DKHCR66UDLX3Z4NKIGQLMA5XMAKPX5ASLRZKND3LKCRZBVGQGUNA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062095731 (1h47m39.843029113s) [noise=true , address=104.36.82.54:4158 , piece_id=JGI64PYCJMQKGBGJEPTEGEWSO4B2QADCF5O34Y77VBVMYL26472Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062459584 (1h47m39.843392966s) [noise=true , address=45.91.116.5:4310 , piece_id=LZ4V7S4QBRX3AGHCTBNHVFCJWYSRTHVASBCLNRW6UKJZRZMPFD2Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062862004 (1h47m39.843795387s) [noise=true , address=45.91.116.9:4273 , piece_id=QPA3YEG3VWDEDXKJUHAA77TBPV7VIEQKWLLIIT4NRX7JETDSSOOQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.063252426 (1h47m39.844185809s) [noise=true , address=77.90.31.11:4558 , piece_id=VBLQN6C6F5NJNUX533PZ6XG2Q74352NBWICFCYBSJA53VC4GXA4A, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.063639960 (1h47m39.844573341s) [noise=true , address=79.127.219.38:4015 , piece_id=DHD4PWLFEUT5Y27KTMGFCAUCHQNSM4VXJZLDS27LMKXMPQRF7V2Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.064095102 (1h47m39.845028485s) [noise=true , address=79.127.205.245:4249 , piece_id=YUDB257EPXWX5OLB62WWWVXQEPYYR2RZRW6PHUCJXCPA6GKFZXWA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.064477662 (1h47m39.84541103s) [noise=true , address=104.36.82.47:4530 , piece_id=AWIXTDPD44XGNJ4SGIL5GLUQDBEWZQLR4RMMLQUSPKRFGXBRG7ZA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.125017691 (1h47m39.905951054s) [upload=1648]: Next returned nil segment Jul 26 23:39:24.125039909 (1h47m39.905973295s) [upload=1648]: Waiting for error group managing segments... Jul 26 23:39:24.129407680 (1h47m39.910341068s) [noise=true , address=79.127.219.38:4015 , piece_id=DHD4PWLFEUT5Y27KTMGFCAUCHQNSM4VXJZLDS27LMKXMPQRF7V2Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.168785826 (1h47m39.949719201s) [noise=true , address=77.90.31.12:4614 , piece_id=SJ5V3WFJVN3376OE4AIOLRUUWTAGJ3IHZM2WHBQ6D5PMLFMSA4LA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.657921679 (1h47m40.438855055s) [noise=true , address=79.127.205.233:4640 , piece_id=22M2A6452J6XA3JFCE3C6GGYNXHDLT7EYTBDIPICAGTY2DDSKFOA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.662741356 (1h47m40.443674723s) [noise=true , address=79.127.205.234:4618 , piece_id=Z2QWR75CR7PO6LNLQKV62H4WQGAI26BKLTZN3GSXE2PT5XLHG6LA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.991068588 (1h47m40.772001964s) [noise=true , address=79.127.205.235:4691 , piece_id=DGWB6WXMIOYMYYD6S3KH52UZMQVXNUQB5CGODP47O4IM77MJHYUQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:27.062691589 (1h47m42.843624956s) [noise=true , address=45.91.116.11:4121 , piece_id=QHOLWAUP4JJ2P4FSHOPIXQJFR2J67LZIKJJM4W7DV6WJG6JMLDIQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:29.507782069 (1h47m45.288715437s) [noise=true , address=79.127.205.245:4249 , piece_id=YUDB257EPXWX5OLB62WWWVXQEPYYR2RZRW6PHUCJXCPA6GKFZXWA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:30.163105689 (1h47m45.944039054s) [noise=true , address=45.91.116.9:4273 , piece_id=QPA3YEG3VWDEDXKJUHAA77TBPV7VIEQKWLLIIT4NRX7JETDSSOOQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:31.510364639 (1h47m47.291298001s) [noise=true , address=45.91.116.5:4310 , piece_id=LZ4V7S4QBRX3AGHCTBNHVFCJWYSRTHVASBCLNRW6UKJZRZMPFD2Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:36.449125481 (1h47m52.230058849s) [noise=true , address=79.127.201.215:4353 , piece_id=2BMTPA7JVRDZCZUXEN46YW2GV4KJP4VUTXXWCKSADVFBS43V75PA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:40:02.128987889 (1h48m17.909921269s) [noise=true , address=77.90.31.11:4558 , piece_id=VBLQN6C6F5NJNUX533PZ6XG2Q74352NBWICFCYBSJA53VC4GXA4A, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:23.990505710 (1h48m39.771439098s) [noise=true , address=104.36.82.52:4092 , piece_id=SM24GEERL7EHLUPKZ422HP75G3U5DGV76KBO27K2TZJESKA7JOKA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:37472->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.990732371 (1h48m39.77166575s) [noise=true , address=104.36.82.54:4158 , piece_id=JGI64PYCJMQKGBGJEPTEGEWSO4B2QADCF5O34Y77VBVMYL26472Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:40:23.991012588 (1h48m39.771945962s) [noise=true , address=104.36.82.46:4492 , piece_id=JMGB57YRCYCU6ADCVVDG5ELOEZMMRTB3BF7Q3ODKOHHDAH53BK4A, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: write tcp 159.223.143.39:56314->104.36.82.46:4492: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991174828 (1h48m39.772108201s) [noise=true , address=104.36.82.47:4530 , piece_id=AWIXTDPD44XGNJ4SGIL5GLUQDBEWZQLR4RMMLQUSPKRFGXBRG7ZA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991224739 (1h48m39.772158108s) [noise=true , address=104.36.82.55:4207 , piece_id=YVFUZ2LA4OA3AFJGEY7X3ZVRLZZXM3WNFSFVBOQ4BT4TJOF2OQEA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:59898->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991288266 (1h48m39.772221641s) [noise=true , address=104.36.82.22:4537 , piece_id=7STK4LSTI6IWOWREQNSKQVH5ZGSQU3IBCJJFQQTPITK6HKB4GMGQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y, address:104.36.82.22:4537): protocol: write tcp 159.223.143.39:58114->104.36.82.22:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991443004 (1h48m39.77237637s) [noise=true , address=77.90.31.21:4912 , piece_id=DKHCR66UDLX3Z4NKIGQLMA5XMAKPX5ASLRZKND3LKCRZBVGQGUNA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:40:23.991479303 (1h48m39.772412688s) [seg_pos={0 0}, upload=1648]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:40:24.017354981 (1h48m39.798288347s) [upload=1648]: Done uploading segments... Jul 26 23:40:24.017385598 (1h48m39.798318996s) [upload=1648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.028554508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.050078305s FAIL (2025-07-26 23:40:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== : Starting upload Jul 26 23:39:53.651659456 (1h48m9.432592821s) [upload=1656]: Done starting upload Jul 26 23:39:53.651795715 (1h48m9.432729094s) [upload=1656]: Uploading segments... Jul 26 23:39:53.652022529 (1h48m9.432955897s) [upload=1656]: Got next segment. Inline: false Jul 26 23:39:53.652177070 (1h48m9.433110436s) [upload=1656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:53.652184134 (1h48m9.433117509s) [upload=1656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:53.760344617 (1h48m9.541278013s) [seg_pos={0 0}, upload=1656]: Begin upload segment... Jul 26 23:39:53.760424673 (1h48m9.541358039s) [seg_pos={0 0}, upload=1656]: Done begin upload segment. Jul 26 23:39:53.760459139 (1h48m9.541392526s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.760511419 (1h48m9.5414448s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761043421 (1h48m9.541976797s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761155851 (1h48m9.542089232s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761542565 (1h48m9.542475946s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761742109 (1h48m9.542675495s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762101713 (1h48m9.543035094s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762304875 (1h48m9.543238262s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762586015 (1h48m9.543519392s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762908490 (1h48m9.543841879s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763217573 (1h48m9.544150957s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763475029 (1h48m9.544408418s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763719412 (1h48m9.544652792s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763976761 (1h48m9.544910146s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764165834 (1h48m9.54509922s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764445958 (1h48m9.545379346s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.857635650 (1h48m9.638569016s) [upload=1656]: Next returned nil segment Jul 26 23:39:53.857653813 (1h48m9.638587183s) [upload=1656]: Waiting for error group managing segments... Jul 26 23:39:53.861881868 (1h48m9.642815249s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.861935675 (1h48m9.642869038s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862515297 (1h48m9.643448675s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862570367 (1h48m9.643503736s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.863057776 (1h48m9.643991146s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.866234319 (1h48m9.6471677s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.985553629 (1h48m9.766486998s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:40:23.614501940 (1h48m39.395435321s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:46296->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614816596 (1h48m39.395749978s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614954894 (1h48m39.395888279s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615058497 (1h48m39.39599188s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615118572 (1h48m39.396051953s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615198214 (1h48m39.396131594s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615286224 (1h48m39.3962196s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615304725 (1h48m39.396238092s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615342159 (1h48m39.396275535s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615382082 (1h48m39.396315457s) [seg_pos={0 0}, upload=1656]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:40:23.662236247 (1h48m39.443169685s) [upload=1656]: Done uploading segments... Jul 26 23:40:23.662284995 (1h48m39.443218371s) [upload=1656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.050121088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [...........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.028612831s FAIL (2025-07-26 23:40:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.028640194s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........XX...X...X..XX.X...X..X.......X...X....XX..........] duration: 39.060684706s FAIL (2025-07-26 23:39:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.060703072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053780697s FAIL (2025-07-26 23:39:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.053796046s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX] duration: 30.000964872s FAIL (2025-07-26 23:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001009039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034295604s FAIL (2025-07-26 23:39:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== _SegmentBegin Jul 26 23:38:47.421931797 (1h47m3.202865179s) [seg_pos={0 0}, upload=1642]: Begin upload segment... Jul 26 23:38:47.421982422 (1h47m3.202915802s) [seg_pos={0 0}, upload=1642]: Done begin upload segment. Jul 26 23:38:47.422019079 (1h47m3.20295247s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422027887 (1h47m3.202961269s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422501337 (1h47m3.20343472s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422512233 (1h47m3.203445615s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422898572 (1h47m3.203831956s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423379839 (1h47m3.204313217s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423781818 (1h47m3.204715198s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424278599 (1h47m3.205211978s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424676105 (1h47m3.205609485s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425058266 (1h47m3.205991648s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425439306 (1h47m3.206372674s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425806572 (1h47m3.206739949s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426180477 (1h47m3.207113862s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426580680 (1h47m3.207514049s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.427600434 (1h47m3.208533815s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.428074992 (1h47m3.209008366s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.499282369 (1h47m3.280215738s) [upload=1642]: Next returned nil segment Jul 26 23:38:47.499299554 (1h47m3.280232934s) [upload=1642]: Waiting for error group managing segments... Jul 26 23:38:47.505148623 (1h47m3.286082s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505300642 (1h47m3.286234012s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505694113 (1h47m3.286627476s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.658447490 (1h47m3.439380859s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.953784843 (1h47m3.734718214s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:39:17.347362137 (1h47m33.128295532s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347615763 (1h47m33.128549139s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347692149 (1h47m33.128625575s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:60548->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347747760 (1h47m33.12868114s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 159.223.143.39:59656->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347812687 (1h47m33.128746075s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347869793 (1h47m33.128803169s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347928947 (1h47m33.128862311s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347973537 (1h47m33.128906913s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347985423 (1h47m33.128918794s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348010236 (1h47m33.128943603s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348163130 (1h47m33.129096495s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348240417 (1h47m33.129173795s) [seg_pos={0 0}, upload=1642]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:39:17.380145118 (1h47m33.161078487s) [upload=1642]: Done uploading segments... Jul 26 23:39:17.380169611 (1h47m33.161102993s) [upload=1642]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.034315477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........XX...X...X..XX.X...X..X.......X...X....XX.........] duration: 52.011418003s FAIL (2025-07-26 23:39:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.011453342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.048085333s FAIL (2025-07-26 23:39:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m_ObjectBegin Jul 26 23:38:10.284561719 (1h46m26.065495097s) [upload=1628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:10.321465178 (1h46m26.102398563s) [seg_pos={0 0}, upload=1628]: Begin upload segment... Jul 26 23:38:10.321530867 (1h46m26.102464232s) [seg_pos={0 0}, upload=1628]: Done begin upload segment. Jul 26 23:38:10.321704631 (1h46m26.102638011s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.321765633 (1h46m26.102699017s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.322215816 (1h46m26.103149198s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323171821 (1h46m26.104105208s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323866994 (1h46m26.10480038s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.324688778 (1h46m26.105622162s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.325242287 (1h46m26.106175665s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326044037 (1h46m26.106977422s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326676184 (1h46m26.107609567s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327510134 (1h46m26.108443522s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327944372 (1h46m26.108877754s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328431350 (1h46m26.109364735s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328707695 (1h46m26.109641079s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329331982 (1h46m26.110265367s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329445281 (1h46m26.110378667s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329904045 (1h46m26.110837431s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.330629252 (1h46m26.111562635s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.331398742 (1h46m26.112332129s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.415942595 (1h46m26.196875972s) [upload=1628]: Next returned nil segment Jul 26 23:38:10.415969697 (1h46m26.196903072s) [upload=1628]: Waiting for error group managing segments... Jul 26 23:38:10.423421258 (1h46m26.20435464s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.450413342 (1h46m26.231346723s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.507513144 (1h46m26.288446509s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.635830264 (1h46m26.416763631s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.885921183 (1h46m26.666854544s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.905141234 (1h46m26.686074603s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.945685357 (1h46m26.726618725s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:11.345380090 (1h46m27.126313466s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:14.304790376 (1h46m30.085723752s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:38:14.944937304 (1h46m30.725870692s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:20.402234579 (1h46m36.183167948s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:26.046187863 (1h46m41.827121239s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:39:10.252609249 (1h47m26.033542629s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252723229 (1h47m26.033656605s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252804093 (1h47m26.033737469s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252882501 (1h47m26.03381588s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:104.36.82.21:4474): protocol: rpc: dial tcp 104.36.82.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252940420 (1h47m26.033873797s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1qxsiscygr1N83dQS4UE53NJSRcLKsAtuDRCuk7ZXZQ99Yswhi, address:104.36.82.20:4453): protocol: rpc: dial tcp 104.36.82.20:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253019370 (1h47m26.033952748s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:59284->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253058304 (1h47m26.033991686s) [seg_pos={0 0}, upload=1628]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:39:10.297369201 (1h47m26.07830257s) [upload=1628]: Done uploading segments... Jul 26 23:39:10.297408551 (1h47m26.078341925s) [upload=1628]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.048114006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [.............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026301846s FAIL (2025-07-26 23:39:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026330682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032727023s FAIL (2025-07-26 23:38:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 26 23:38:17.282381672 (1h46m33.063315054s) [upload=1630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:38:17.282388081 (1h46m33.063321456s) [upload=1630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:17.325457222 (1h46m33.106390598s) [seg_pos={0 0}, upload=1630]: Begin upload segment... Jul 26 23:38:17.325499025 (1h46m33.10643239s) [seg_pos={0 0}, upload=1630]: Done begin upload segment. Jul 26 23:38:17.325522894 (1h46m33.106456275s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326015952 (1h46m33.106949334s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326503029 (1h46m33.10743641s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326863575 (1h46m33.10779695s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.327943710 (1h46m33.108877095s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.328763942 (1h46m33.109697321s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329450902 (1h46m33.110384285s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329827291 (1h46m33.110760669s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329914440 (1h46m33.110847824s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330382271 (1h46m33.111315649s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330842660 (1h46m33.111776042s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331249859 (1h46m33.112183237s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331700245 (1h46m33.112633631s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332193010 (1h46m33.113126396s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332877279 (1h46m33.113810664s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.333406079 (1h46m33.114339466s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.415164740 (1h46m33.19609812s) [upload=1630]: Next returned nil segment Jul 26 23:38:17.415188443 (1h46m33.19612183s) [upload=1630]: Waiting for error group managing segments... Jul 26 23:38:17.419552774 (1h46m33.200486142s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420179959 (1h46m33.201113325s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420724133 (1h46m33.201657498s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421161211 (1h46m33.202094586s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421636875 (1h46m33.202570244s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.483748356 (1h46m33.264681732s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:47.247103928 (1h47m3.028037302s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247174663 (1h47m3.028108028s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247208890 (1h47m3.02814228s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247224212 (1h47m3.028157578s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247274888 (1h47m3.028208253s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247297158 (1h47m3.028230534s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247325516 (1h47m3.028258881s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42306->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247363193 (1h47m3.028296558s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247388224 (1h47m3.028321593s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247425257 (1h47m3.02835863s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:40042->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247465754 (1h47m3.028399139s) [seg_pos={0 0}, upload=1630]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:38:47.277303759 (1h47m3.05823714s) [upload=1630]: Done uploading segments... Jul 26 23:38:47.277332606 (1h47m3.058265987s) [upload=1630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.03275199s core uplink us-select-1

1
...
6761
6762
6763
6764
6765
...
7257