Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 546.386073ms FAIL (2025-07-26 22:32:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.415953ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 207.121511ms FAIL (2025-07-26 22:32:37 UTC) orbiter warning: uplink: too many requests Duration: 207.155143ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.] duration: 5.73014305s FAIL (2025-07-26 22:32:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.730167129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X........X.X.X...XX..] duration: 4.834799903s FAIL (2025-07-26 22:32:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.834834902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 1m0.030676676s FAIL (2025-07-26 22:32:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.03072904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 247.309138ms FAIL (2025-07-26 22:32:05 UTC) orbiter warning: uplink: too many requests Duration: 247.346203ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........XX..XXX..X..XX..X..XX.X....X....X........X.......] duration: 641.280149ms FAIL (2025-07-26 22:31:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 641.315695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..X........XXX.XXX.X.XX..X..X......X.............X.....] duration: 541.603125ms FAIL (2025-07-26 22:31:54 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 541.63824ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 587.336207ms FAIL (2025-07-26 22:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 587.418487ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 1m0.029674709s FAIL (2025-07-26 22:31:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Jul 26 22:30:23.175715785 (38m38.956649175s) [upload=602]: Starting upload Jul 26 22:30:23.176643641 (38m38.957577037s) [upload=602]: Done starting upload Jul 26 22:30:23.176858032 (38m38.957791403s) [upload=602]: Uploading segments... Jul 26 22:30:23.177442532 (38m38.958375909s) [upload=602]: Got next segment. Inline: false Jul 26 22:30:23.177529730 (38m38.958463096s) [upload=602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:30:23.177539149 (38m38.958472517s) [upload=602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:30:23.220426626 (38m39.001360014s) [seg_pos={0 0}, upload=602]: Begin upload segment... Jul 26 22:30:23.220496207 (38m39.001429595s) [seg_pos={0 0}, upload=602]: Done begin upload segment. Jul 26 22:30:23.220541622 (38m39.001475014s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.221481322 (38m39.002414711s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222101054 (38m39.003034443s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222767744 (38m39.003701132s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223371716 (38m39.004305103s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223841406 (38m39.00477479s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224429753 (38m39.005363139s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224883320 (38m39.005816708s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.225319227 (38m39.006252616s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.226710795 (38m39.007644183s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227512914 (38m39.00844641s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227953319 (38m39.008886708s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.228555621 (38m39.00948901s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229094745 (38m39.010028132s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229653467 (38m39.010586846s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230260750 (38m39.011194139s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230834024 (38m39.011767412s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.231466514 (38m39.012399898s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.358902366 (38m39.139835729s) [upload=602]: Next returned nil segment Jul 26 22:30:23.358921206 (38m39.139854582s) [upload=602]: Waiting for error group managing segments... Jul 26 22:30:23.396711622 (38m39.177644997s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.429135315 (38m39.210068678s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.492792568 (38m39.273725937s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.786085237 (38m39.567018615s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:23.833627956 (38m39.614561334s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.850607137 (38m39.631540513s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.963675075 (38m39.744608451s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.037010439 (38m39.817943809s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.192023890 (38m39.972957259s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.994113890 (38m40.775047251s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:26.699149799 (38m42.480083167s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.523850606 (38m44.304783973s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.684871960 (38m44.465805324s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:31.099634536 (38m46.880567918s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:31.341563839 (38m47.122497218s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:54.205996972 (39m9.986930349s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:31:23.146066980 (39m38.927000363s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146140712 (39m38.927074091s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:45828->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146194450 (39m38.927127835s) [seg_pos={0 0}, upload=602]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:31:23.172654781 (39m38.953588151s) [upload=602]: Done uploading segments... Jul 26 22:31:23.172680200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029725569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................................X........X.X.X...XX.] duration: 4.987162153s FAIL (2025-07-26 22:31:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.987183675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X........X.X.X...XX] duration: 1m0.00063588s FAIL (2025-07-26 22:31:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000652214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............XX..XXX..X..XX..X..XX.X....X....X........X......] duration: 1.114872713s FAIL (2025-07-26 22:31:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.114906933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 231.383667ms FAIL (2025-07-26 22:31:00 UTC) orbiter warning: uplink: too many requests Duration: 231.420682ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X........XXX.XXX.X.XX..X..X......X.............X....] duration: 452.889304ms FAIL (2025-07-26 22:30:51 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 452.954895ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 195.576028ms FAIL (2025-07-26 22:30:37 UTC) orbiter warning: uplink: too many requests Duration: 195.607825ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.84352625s FAIL (2025-07-26 22:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.843580658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X..X........XXX.XXX.X.XX..X..X......X.............X...] duration: 450.840526ms FAIL (2025-07-26 22:30:31 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 450.882768ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............XX..XXX..X..XX..X..XX.X....X....X........X.....] duration: 562.993517ms FAIL (2025-07-26 22:30:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.175298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X..X........XXX.XXX.X.XX..X..X......X.............X..] duration: 515.983574ms FAIL (2025-07-26 22:30:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 516.038922ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X........X.X.X...X] duration: 1m0.001177122s FAIL (2025-07-26 22:30:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001276067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 1m0.0392517s FAIL (2025-07-26 22:30:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== DRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872330524 (37m24.653263907s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872852006 (37m24.653785395s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872977116 (37m24.653910502s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873348563 (37m24.654281945s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873460795 (37m24.654394183s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873814983 (37m24.654748371s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873904265 (37m24.654837652s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874272710 (37m24.655206097s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874350404 (37m24.655283792s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874729933 (37m24.655663317s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874798536 (37m24.65573192s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875192974 (37m24.656126364s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875280310 (37m24.656213703s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875649341 (37m24.656582728s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875758061 (37m24.656691447s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.876344348 (37m24.657277726s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:09.139975922 (37m24.920909285s) [upload=582]: Next returned nil segment Jul 26 22:29:09.140138981 (37m24.921072343s) [upload=582]: Waiting for error group managing segments... Jul 26 22:29:09.474617798 (37m25.255551165s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.476801572 (37m25.25773494s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.561523995 (37m25.342457365s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.507316685 (37m27.288250074s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.941822852 (37m27.722756237s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:12.834296579 (37m28.615229957s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:13.872012428 (37m29.652945807s) [noise=true , address=45.91.116.10:4100 , piece_id=VPHKMJGRLKOKDRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:19.175006841 (37m34.955940211s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:21.249247070 (37m37.030180453s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:04.345974914 (38m20.126908287s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:30:08.725951783 (38m24.506885162s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726063777 (38m24.506997155s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1eRovJAp3AyUHP4qkuDvSSGYb9TG6U4tSNzniGJBeYLhkmiUVy, address:104.36.82.41:4779): protocol: rpc: dial tcp 104.36.82.41:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726143819 (38m24.507077204s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726212930 (38m24.507146312s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726277487 (38m24.507210863s) [noise=true , address=104.36.82.56:4233 , piece_id=56UZRSJOWFMOUEGYU4CFLN4O5I4HHJ7JRPEULXON2XYAGROXDZPQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726339540 (38m24.507272924s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: rpc: dial tcp 104.36.82.42:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726402211 (38m24.507335589s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12B6ymfqNvqiDsAbkF8H4ExxEuYaYb7gYQ45Y5GF4PGAAwmjbyd, address:104.36.82.45:4928): protocol: rpc: dial tcp 104.36.82.45:4928: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726467024 (38m24.50740041s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12tD1Zfi4ZxujF9pVdgW5k2wnejXsa7tf1u25FuucRXeneGP3dz, address:104.36.82.36:4651): protocol: write tcp 159.223.143.39:43616->104.36.82.36:4651: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726489768 (38m24.507423159s) [seg_pos={0 0}, upload=582]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039269014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X........XXX.XXX.X.XX..X..X......X.............X.] duration: 2.001677008s FAIL (2025-07-26 22:29:48 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 2.001717878s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 515.273029ms FAIL (2025-07-26 22:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.306952ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 221.064152ms FAIL (2025-07-26 22:29:41 UTC) orbiter warning: uplink: too many requests Duration: 221.097603ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 220.865789ms FAIL (2025-07-26 22:29:24 UTC) orbiter warning: uplink: too many requests Duration: 220.900138ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............XX..XXX..X..XX..X..XX.X....X....X........X....] duration: 456.197031ms FAIL (2025-07-26 22:29:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.232604ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 611.778316ms FAIL (2025-07-26 22:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.807268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X........XXX.XXX.X.XX..X..X......X.............X] duration: 405.750901ms FAIL (2025-07-26 22:29:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 405.782476ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 1m0.03076395s FAIL (2025-07-26 22:29:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== eg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.781832705 (36m24.562766094s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782330540 (36m24.563264048s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782464497 (36m24.563397886s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782878492 (36m24.563811878s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783111638 (36m24.564045026s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783530994 (36m24.56446438s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783765643 (36m24.56469903s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783946386 (36m24.564879778s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784412344 (36m24.565345728s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784460953 (36m24.565394339s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784942001 (36m24.565875392s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786106520 (36m24.56703991s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786393751 (36m24.567327141s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786632451 (36m24.567565839s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786826948 (36m24.567760337s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.787078509 (36m24.568011888s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.869617862 (36m24.650551257s) [upload=568]: Next returned nil segment Jul 26 22:28:08.869639727 (36m24.650573114s) [upload=568]: Waiting for error group managing segments... Jul 26 22:28:08.903844258 (36m24.684777623s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:08.997481088 (36m24.778414463s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.125047128 (36m24.905980512s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.283982565 (36m25.064915935s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.389541409 (36m25.170474777s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.477572185 (36m25.258505554s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:14.011496993 (36m29.792430379s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:22.943945444 (36m38.724878812s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:29:08.617901143 (37m24.39883452s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fDTUq74fpQVaFNsfxPmKwMpG9eUfFDcjPd5FutuqXiBp9R5EN, address:104.36.82.18:4393): protocol: rpc: dial tcp 104.36.82.18:4393: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.617992630 (37m24.398925992s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: rpc: dial tcp 104.36.82.7:4916: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618231339 (37m24.399164717s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: rpc: dial tcp 104.36.82.11:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618318480 (37m24.399251863s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618394683 (37m24.39932806s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:34384->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618462310 (37m24.399395687s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618471039 (37m24.399404419s) [noise=true , address=104.36.82.52:4094 , piece_id=YFYPN3FRCIHUN22VGHWTK53JTQKDGPPVQVAWQGYORBLQWDKN7XQA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618560947 (37m24.399494325s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618599925 (37m24.399533288s) [noise=true , address=104.36.82.12:4152 , piece_id=J4Y7ZDRMQTXU6IVCFEC7U4JV23HQA47CYVQONKP62ASLXN6JYFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 159.223.143.39:58020->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618618191 (37m24.399551572s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: write tcp 159.223.143.39:35046->104.36.82.40:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618677546 (37m24.399610934s) [seg_pos={0 0}, upload=568]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:29:08.644975149 (37m24.425908526s) [upload=568]: Done uploading segments... Jul 26 22:29:08.645009948 (37m24.425943394s) [upload=568]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030874016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 516.687863ms FAIL (2025-07-26 22:28:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.70933ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............XX..XXX..X..XX..X..XX.X....X....X........X...] duration: 453.101199ms FAIL (2025-07-26 22:28:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.134146ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 51.546106ms FAIL (2025-07-26 22:28:45 UTC) orbiter warning: uplink: too many requests Duration: 51.572545ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X........X.X.X...] duration: 4.119879275s FAIL (2025-07-26 22:28:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.119902276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 658.894921ms FAIL (2025-07-26 22:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.010631ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 1m0.039342136s FAIL (2025-07-26 22:28:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =555]: Got next segment. Inline: false Jul 26 22:27:08.521554810 (35m24.302488195s) [upload=555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:27:08.521561305 (35m24.302494689s) [upload=555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:27:08.566725134 (35m24.34765863s) [seg_pos={0 0}, upload=555]: Begin upload segment... Jul 26 22:27:08.566868456 (35m24.347801846s) [seg_pos={0 0}, upload=555]: Done begin upload segment. Jul 26 22:27:08.566926374 (35m24.347859781s) [noise=true , address=79.127.201.215:4358 , piece_id=2ZGKHBGSRVFZAXKJEN3GBZCEJVPEIHXZ2ZIAARFJRJRSQQBK5XDA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.566948436 (35m24.347881825s) [noise=true , address=77.90.31.16:4746 , piece_id=5DBORRAQIHTBNXJYJEOOBQC4YOCLPWCG5H3FGYCZAZ5E7BTUXZJQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567073386 (35m24.34800677s) [noise=true , address=45.91.116.5:4310 , piece_id=D4PIYEMA7PFEUESZ7BPNOHPEONWQKBRL2BGWIHG366PFV77MPGOA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567519381 (35m24.348452772s) [noise=true , address=64.49.13.138:4483 , piece_id=5WKBRLGZZV43P3AMSSUVRWWIYSF54MYZZ33AVQVV463WLYBGP26A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567626007 (35m24.34855939s) [noise=true , address=104.36.82.52:4090 , piece_id=DC6RDKRDWHE3GI7VCMOGHV5YH5ACOJ5UQRWNXJ46IVYEBSLPHG5Q, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567987293 (35m24.348920672s) [noise=true , address=79.127.205.233:4639 , piece_id=EBVCFHT5G2LHKS2W6WTQAKPXXS3PEESY22MJFB736KG6LHCQY7VQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568455886 (35m24.349389271s) [noise=true , address=79.127.205.246:4285 , piece_id=RB4TDSIXLN4VAQEXQEARJFDF24MO4EOMB756URVLHOZ3CRAAJQBQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568561311 (35m24.3494947s) [noise=true , address=104.36.82.51:4081 , piece_id=Y3UHTSYTIWFWRSQFFXTJYD3SI4DFCIB3QUWSEQHQAL7N3Y2KKSCQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568996915 (35m24.349930339s) [noise=true , address=104.36.82.4:4827 , piece_id=AOIJVARJ5LRFWLGTP24ZBT2MGQTER2JKQIH4XDO6MKPH4I2OUFAQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569015289 (35m24.349948675s) [noise=true , address=104.36.82.16:4300 , piece_id=SH6UOAO5HJIBRNMA34PQ2YAXYLBVLTMGQ6ULB6IHKP3F4NAYWS7A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569455644 (35m24.350389031s) [noise=true , address=79.127.205.235:4689 , piece_id=HNHDF4CU5R6BGSVI3KWNUHGPNXE25U3GVYZGUXUDCES7CP4OZEOA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569872680 (35m24.350806068s) [noise=true , address=104.36.82.21:4499 , piece_id=PRNLGOYXS3BQJEJAXAO6QJD4O4LQUFQYKZQEJDGKVBF742HJAESA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569989884 (35m24.35092328s) [noise=true , address=45.91.116.14:4222 , piece_id=4YKN5ZIM2N2FV53GO3H7S6JHGIMSN5KDFV3VKHIHCXRMAKILH67A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.570344970 (35m24.351278357s) [noise=true , address=79.127.205.234:4615 , piece_id=CDSJAQOFK7HSEQMST27L4YBTCSOJETH6MDRXT46WQSSRWI3RQLXA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.570820355 (35m24.351753743s) [noise=true , address=79.127.205.245:4243 , piece_id=R5ZNLJJP5JQCWIQ2DLO3IDWJI6GBY3WASG7HRLNTDIXZAFPPJACQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.575996870 (35m24.356930245s) [noise=true , address=104.36.82.56:4234 , piece_id=6E6KBP5GCJHYZIAFEP5J4RWR7TJ5SV5AFSA6FRWLNUSAYAVEYMTQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.578658024 (35m24.359591411s) [noise=true , address=79.127.219.38:4015 , piece_id=WZD2VEADTQZSJYWWQ665VCZNU3CWL3Z5RY2PUSA56MLYNOGZJYLA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.587960422 (35m24.368893809s) [noise=true , address=45.91.116.11:4146 , piece_id=AQPQUYI3YCZC23ALHTLJYZ4YGWZ6XRQMS3PH62F63EV7YASALXHQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.683348272 (35m24.46428164s) [upload=555]: Next returned nil segment Jul 26 22:27:08.683365120 (35m24.464298501s) [upload=555]: Waiting for error group managing segments... Jul 26 22:27:08.750506310 (35m24.531439688s) [noise=true , address=77.90.31.16:4746 , piece_id=5DBORRAQIHTBNXJYJEOOBQC4YOCLPWCG5H3FGYCZAZ5E7BTUXZJQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:08.784020342 (35m24.564953712s) [noise=true , address=79.127.219.38:4015 , piece_id=WZD2VEADTQZSJYWWQ665VCZNU3CWL3Z5RY2PUSA56MLYNOGZJYLA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.151235321 (35m24.932168691s) [noise=true , address=79.127.205.235:4689 , piece_id=HNHDF4CU5R6BGSVI3KWNUHGPNXE25U3GVYZGUXUDCES7CP4OZEOA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.169744980 (35m24.950678355s) [noise=true , address=79.127.205.245:4243 , piece_id=R5ZNLJJP5JQCWIQ2DLO3IDWJI6GBY3WASG7HRLNTDIXZAFPPJACQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.202860077 (35m24.983793443s) [noise=true , address=79.127.205.234:4615 , piece_id=CDSJAQOFK7HSEQMST27L4YBTCSOJETH6MDRXT46WQSSRWI3RQLXA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.211433807 (35m24.992367491s) [noise=true , address=79.127.205.233:4639 , piece_id=EBVCFHT5G2LHKS2W6WTQAKPXXS3PEESY22MJFB736KG6LHCQY7VQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.362863020 (35m25.143796387s) [noise=true , address=79.127.205.246:4285 , piece_id=RB4TDSIXLN4VAQEXQEARJFDF24MO4EOMB756URVLHOZ3CRAAJQBQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:12.882201179 (35m28.663134543s) [noise=true , address=79.127.201.215:4358 , piece_id=2ZGKHBGSRVFZAXKJEN3GBZCEJVPEIHXZ2ZIAARFJRJRSQQBK5XDA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:13.954258922 (35m29.7351923s) [noise=true , address=64.49.13.138:4483 , piece_id=5WKBRLGZZV43P3AMSSUVRWWIYSF54MYZZ33AVQVV463WLYBGP26A, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:16.171379307 (35m31.952312689s) [noise=true , address=45.91.116.11:4146 , piece_id=AQPQUYI3YCZC23ALHTLJYZ4YGWZ6XRQMS3PH62F63EV7YASALXHQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:27:16.935776619 (35m32.716709998s) [noise=true , address=45.91.116.5:4310 , piece_id=D4PIYEMA7PFEUESZ7BPNOHPEONWQKBRL2BGWIHG366PFV77MPGOA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:18.454183596 (35m34.235116964s) [noise=true , address=45.91.116.14:4222 , piece_id=4YKN5ZIM2N2FV53GO3H7S6JHGIMSN5KDFV3VKHIHCXRMAKILH67A, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:28:08.478930656 (36m24.259864043s) [noise=true , address=104.36.82.52:4090 , piece_id=DC6RDKRDWHE3GI7VCMOGHV5YH5ACOJ5UQRWNXJ46IVYEBSLPHG5Q, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479010621 (36m24.259943999s) [noise=true , address=104.36.82.4:4827 , piece_id=AOIJVARJ5LRFWLGTP24ZBT2MGQTER2JKQIH4XDO6MKPH4I2OUFAQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:1VXFTn1urnJ2qkUcDc5785hJ8L6bNVQTtgcgNeWLaZ9wEkKYVx, address:104.36.82.4:4827): protocol: rpc: dial tcp 104.36.82.4:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479154384 (36m24.260087765s) [noise=true , address=104.36.82.21:4499 , piece_id=PRNLGOYXS3BQJEJAXAO6QJD4O4LQUFQYKZQEJDGKVBF742HJAESA, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:1218y8DtJNTTu94dbqGhywmHR4s99zcz9issmMmpgwusr2KgZxc, address:104.36.82.21:4499): protocol: rpc: dial tcp 104.36.82.21:4499: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479228428 (36m24.260161806s) [noise=true , address=104.36.82.56:4234 , piece_id=6E6KBP5GCJHYZIAFEP5J4RWR7TJ5SV5AFSA6FRWLNUSAYAVEYMTQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479293591 (36m24.260226969s) [noise=true , address=104.36.82.51:4081 , piece_id=Y3UHTSYTIWFWRSQFFXTJYD3SI4DFCIB3QUWSEQHQAL7N3Y2KKSCQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: rpc: dial tcp 104.36.82.51:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479352354 (36m24.260285732s) [noise=true , address=104.36.82.16:4300 , piece_id=SH6UOAO5HJIBRNMA34PQ2YAXYLBVLTMGQ6ULB6IHKP3F4NAYWS7A, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: write tcp 159.223.143.39:37764->104.36.82.16:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479375580 (36m24.260308966s) [seg_pos={0 0}, upload=555]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:28:08.514602766 (36m24.295536137s) [upload=555]: Done uploading segments... Jul 26 22:28:08.514630796 (36m24.295564173s) [upload=555]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.039363616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........X..X........XXX.XXX.X.XX..X..X......X.............] duration: 658.414768ms FAIL (2025-07-26 22:28:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 658.447023ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................X........X.X.X..] duration: 5.359646695s FAIL (2025-07-26 22:27:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.359664834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................XX..XXX..X..XX..X..XX.X....X....X........X..] duration: 417.475794ms FAIL (2025-07-26 22:27:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.519671ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 520.561921ms FAIL (2025-07-26 22:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.653861ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 1m0.028741691s FAIL (2025-07-26 22:27:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 24.25156866s) [noise=true , address=64.49.13.155:4916 , piece_id=CKNLNHNRKHRWSUX2DC6C6UBW5W5EXF4JEVDG4N72YRGWPX6II4TA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.471225286 (34m24.252158674s) [noise=true , address=104.36.82.41:4765 , piece_id=LGW6BM4RIHV4ED5ADJFTCHHM7YJBPCE4KWDWU2HK2MAYKTIGN6JQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.471739427 (34m24.252672815s) [noise=true , address=104.36.82.52:4087 , piece_id=7ITOZLFRY63TNYADJFUMFJRONSCPLRTYC5W67SJ4YCBTB6N36W5A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.472277321 (34m24.253210706s) [noise=true , address=104.36.82.16:4317 , piece_id=AFZUMZTZHG23ESTJKPHJVVZWPRCLZZ4LQPB6SHLND6MP4U3T63VA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.474588396 (34m24.255521784s) [noise=true , address=104.36.82.35:4589 , piece_id=PG4SK5QUQ6C6DIM3ROPRU5SOHFHEKSDVSHMMRYVWQ7POOQOIJQPQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475071564 (34m24.256004954s) [noise=true , address=45.91.116.14:4222 , piece_id=7BZVGGSCHKIADBLFGRKTMDDLQM7SRKHASMWGXJGA3BMXRKPJIYHQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475509276 (34m24.256442666s) [noise=true , address=79.127.205.245:4239 , piece_id=PWZQZMIEERREFGK2X7PQLM3HAT6NEVR5WK44WDMC7FTKEY6EY24Q, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475965302 (34m24.256898687s) [noise=true , address=104.36.82.37:4677 , piece_id=A6VF5UCK4DJCKPGPLD5PNR4FS5XFR4NUYB2GJIYA5VXSGBYS23YA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.476394670 (34m24.257328054s) [noise=true , address=104.36.82.55:4228 , piece_id=KTKVQNLA3HRFHOC7VCAY36UKXJW57MDYSLJCWWDDRNETZ4K5QD4A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.476801074 (34m24.25773446s) [noise=true , address=79.127.205.235:4689 , piece_id=N2KTACUFHS6YKN6IO564SVYKX52LXQC2NDP3IBCAM3KWIQE6UHSQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477196730 (34m24.258130122s) [noise=true , address=79.127.205.234:4614 , piece_id=UUJ7SNHI3ARET6F5RSH4I4VQIDCX2B5J2EWTNRADVUUPM4MP2BWA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477610913 (34m24.258544302s) [noise=true , address=45.91.116.9:4267 , piece_id=HLYAB64GJZPQCUCWM37Y57DCWCNAQB4E4HUQ3SIFLNBN2BVQT6QA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477993168 (34m24.258926556s) [noise=true , address=79.127.220.101:4048 , piece_id=AJWNXMV7SFAKXM4VS75F725SWYERPFRUUDGGF6LNA56M6XHSVXJQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.478453454 (34m24.259386843s) [noise=true , address=104.36.82.56:4235 , piece_id=WFY35ZFBT7ZXWR2SDTTDPFCW7QAICXI3AGD5PXNAK5JUG7J3TZCA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.480710981 (34m24.261644369s) [noise=true , address=104.36.82.5:4872 , piece_id=NHTYHU3X2UMBJJEPK7VMQOKLBLTFLRDSOU52KYXQ62IITAKDBS5A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.481248617 (34m24.262182007s) [noise=true , address=77.90.31.17:4796 , piece_id=44LDCR6YJGFZPMBCVYOFTKCEH5ZN2JGHRTCQ2UHMHR6TF5D4HE2Q, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.638997039 (34m24.419930419s) [noise=true , address=104.36.82.55:4228 , piece_id=KTKVQNLA3HRFHOC7VCAY36UKXJW57MDYSLJCWWDDRNETZ4K5QD4A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:26:08.649206875 (34m24.430140266s) [upload=538]: Next returned nil segment Jul 26 22:26:08.649229271 (34m24.430162635s) [upload=538]: Waiting for error group managing segments... Jul 26 22:26:08.962307497 (34m24.743240867s) [noise=true , address=79.127.220.101:4048 , piece_id=AJWNXMV7SFAKXM4VS75F725SWYERPFRUUDGGF6LNA56M6XHSVXJQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.110679643 (34m24.89161301s) [noise=true , address=79.127.205.234:4614 , piece_id=UUJ7SNHI3ARET6F5RSH4I4VQIDCX2B5J2EWTNRADVUUPM4MP2BWA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.206468649 (34m24.987402024s) [noise=true , address=64.49.13.155:4916 , piece_id=CKNLNHNRKHRWSUX2DC6C6UBW5W5EXF4JEVDG4N72YRGWPX6II4TA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.385442736 (34m25.166376114s) [noise=true , address=79.127.205.235:4689 , piece_id=N2KTACUFHS6YKN6IO564SVYKX52LXQC2NDP3IBCAM3KWIQE6UHSQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.673810066 (34m25.454743447s) [noise=true , address=79.127.205.245:4239 , piece_id=PWZQZMIEERREFGK2X7PQLM3HAT6NEVR5WK44WDMC7FTKEY6EY24Q, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:11.217259182 (34m26.998192544s) [noise=true , address=45.91.116.9:4267 , piece_id=HLYAB64GJZPQCUCWM37Y57DCWCNAQB4E4HUQ3SIFLNBN2BVQT6QA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:17.945117750 (34m33.726051129s) [noise=true , address=45.91.116.14:4222 , piece_id=7BZVGGSCHKIADBLFGRKTMDDLQM7SRKHASMWGXJGA3BMXRKPJIYHQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:27:08.393443389 (35m24.174376773s) [noise=true , address=104.36.82.16:4317 , piece_id=AFZUMZTZHG23ESTJKPHJVVZWPRCLZZ4LQPB6SHLND6MP4U3T63VA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1FdJoeW4vejiQx1QkQ1nCkApgQaW6fB66d2usAYVfFuK1PPmRP, address:104.36.82.16:4317): protocol: rpc: dial tcp 104.36.82.16:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393544825 (35m24.174478209s) [noise=true , address=104.36.82.5:4872 , piece_id=NHTYHU3X2UMBJJEPK7VMQOKLBLTFLRDSOU52KYXQ62IITAKDBS5A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: rpc: dial tcp 104.36.82.5:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393625675 (35m24.174559051s) [noise=true , address=104.36.82.52:4087 , piece_id=7ITOZLFRY63TNYADJFUMFJRONSCPLRTYC5W67SJ4YCBTB6N36W5A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393695046 (35m24.174628423s) [noise=true , address=104.36.82.37:4677 , piece_id=A6VF5UCK4DJCKPGPLD5PNR4FS5XFR4NUYB2GJIYA5VXSGBYS23YA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12QgDye4pcKhsndRP7tCxM44KZkSyRnhp219u8JnRHXX7jRutPv, address:104.36.82.37:4677): protocol: rpc: dial tcp 104.36.82.37:4677: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393763323 (35m24.1746967s) [noise=true , address=104.36.82.42:4819 , piece_id=CA3A7OBKHQGMPP44JSEKHXI6KXI7WLN6JONUEFK54KOOMY72CDPQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: rpc: dial tcp 104.36.82.42:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393831356 (35m24.174764734s) [noise=true , address=104.36.82.12:4181 , piece_id=G7XRU7P3CYATUTJTEDYDLW7LAQFRF47X7ABNK25H25SD4XDVK4YQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: rpc: dial tcp 104.36.82.12:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393896920 (35m24.174830297s) [noise=true , address=77.90.31.17:4796 , piece_id=44LDCR6YJGFZPMBCVYOFTKCEH5ZN2JGHRTCQ2UHMHR6TF5D4HE2Q, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393960559 (35m24.174893938s) [noise=true , address=104.36.82.41:4765 , piece_id=LGW6BM4RIHV4ED5ADJFTCHHM7YJBPCE4KWDWU2HK2MAYKTIGN6JQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:104.36.82.41:4765): protocol: rpc: dial tcp 104.36.82.41:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394024616 (35m24.174957993s) [noise=true , address=104.36.82.35:4589 , piece_id=PG4SK5QUQ6C6DIM3ROPRU5SOHFHEKSDVSHMMRYVWQ7POOQOIJQPQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394096792 (35m24.175030176s) [noise=true , address=104.36.82.56:4235 , piece_id=WFY35ZFBT7ZXWR2SDTTDPFCW7QAICXI3AGD5PXNAK5JUG7J3TZCA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 159.223.143.39:47950->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394122351 (35m24.175055739s) [seg_pos={0 0}, upload=538]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:27:08.418086562 (35m24.199019935s) [upload=538]: Done uploading segments... Jul 26 22:27:08.418115856 (35m24.199049233s) [upload=538]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.028770564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 559.755213ms FAIL (2025-07-26 22:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 559.793724ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........X..X........XXX.XXX.X.XX..X..X......X............] duration: 502.187842ms FAIL (2025-07-26 22:26:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 502.212745ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X........X.X.X.] duration: 11.34738458s FAIL (2025-07-26 22:26:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.347413348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X........X.X.X] duration: 1m0.001009426s FAIL (2025-07-26 22:26:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001044754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX..XXX..X..XX..X..XX.X....X....X........X.] duration: 604.067406ms FAIL (2025-07-26 22:26:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 604.101915ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 1m0.033500006s FAIL (2025-07-26 22:26:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== WY5JV5A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242126122 (33m19.023059512s) [noise=true , address=79.127.219.47:4069 , piece_id=IFDIWTMBWAJDEFZFUJCX53VVU2MTJZTE5J5STYCKT6OAFT6UUGKA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242356864 (33m19.023290252s) [noise=true , address=64.49.13.140:4508 , piece_id=7H3YCEUT26RWXLJUV5EQSQRG6SBUX4K5PI3THT7FPYUQXGEZS3CA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242562140 (33m19.023495521s) [noise=true , address=104.36.82.55:4228 , piece_id=FGXP2NNTU5IATOQKF6FHPXTO4A4NA3N7BCIFWM3IQESGGSF64UBQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242912896 (33m19.023846281s) [noise=true , address=79.127.219.38:4004 , piece_id=MSLWHKRKD2ZXPXU73HAEXYU7SESJ4SIEYOLN2CHPUCQYOP3WA5NA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242946351 (33m19.023879741s) [noise=true , address=104.36.82.56:4234 , piece_id=OPYBCBF4AJPAUHMP5UCO4WBM7RJ2RH7ZEHOGAWTDSPGKFKXVLAZA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.243332587 (33m19.024265969s) [noise=true , address=77.90.31.15:4726 , piece_id=3EJGMZIWPWBAVSA6AZPDRO4XNCXDN3QFZNGI5NAJA5DSYHXKRUXQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.243556391 (33m19.024489777s) [noise=true , address=104.36.82.13:4199 , piece_id=2WM67WHZZLFCUU42RYWARGL64IMLNXNL7RNXE7CA75HPCFVLOD6A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.247501553 (33m19.028434941s) [noise=true , address=77.90.31.17:4796 , piece_id=PIDEYKX37B4DDG3S2W5JCUOGXGUSZF57RCADSY3FESNJVIL4MVPA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248246455 (33m19.029179832s) [noise=true , address=77.90.31.11:4560 , piece_id=DCQGCLCXOEVCNF3YTHFRATEAFL47GEJTWRANKIKRDH6TGIYIXIIQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248735549 (33m19.029668935s) [noise=true , address=104.36.82.7:4922 , piece_id=5MLUACJA4KYMIHUHL5M4RF2WSEGORQCPDQTC4X46NZ2QVYQBLK7A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248852573 (33m19.029785961s) [noise=true , address=104.36.82.2:4755 , piece_id=V4EX5DHU2FBKS6PBGHDKVVVWLX2IPHFE3PDVTBLEWU2ANQQ5K27A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.249397978 (33m19.030331365s) [noise=true , address=45.91.116.9:4267 , piece_id=P7A3F5LKHBUZ6IP3WTVLOSWYZ73QK7XSLFMF44YSWPVSTEM2EOGA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.249862061 (33m19.03079545s) [noise=true , address=104.36.82.51:4084 , piece_id=UEE5PVLTDFFAIPEBUZXP5VUILHX5FSNICCREOTDU4MWJIX74JUJA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.250461925 (33m19.031395321s) [noise=true , address=104.36.82.11:4146 , piece_id=TXPTK6Q4Y5ZE4IVEDMNXM4YTTB6CIA3VN7ZUFQQALIE5MIIEP6KA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.335517996 (33m19.116451388s) [noise=true , address=79.127.219.47:4069 , piece_id=IFDIWTMBWAJDEFZFUJCX53VVU2MTJZTE5J5STYCKT6OAFT6UUGKA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:03.362778206 (33m19.143711584s) [upload=522]: Next returned nil segment Jul 26 22:25:03.362802210 (33m19.143735597s) [upload=522]: Waiting for error group managing segments... Jul 26 22:25:03.365336869 (33m19.146270233s) [noise=true , address=79.127.219.38:4004 , piece_id=MSLWHKRKD2ZXPXU73HAEXYU7SESJ4SIEYOLN2CHPUCQYOP3WA5NA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.372034817 (33m19.152968181s) [noise=true , address=45.91.116.9:4267 , piece_id=P7A3F5LKHBUZ6IP3WTVLOSWYZ73QK7XSLFMF44YSWPVSTEM2EOGA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.401661243 (33m19.182594607s) [noise=true , address=77.90.31.11:4560 , piece_id=DCQGCLCXOEVCNF3YTHFRATEAFL47GEJTWRANKIKRDH6TGIYIXIIQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.526068921 (33m19.307002291s) [noise=true , address=77.90.31.15:4726 , piece_id=3EJGMZIWPWBAVSA6AZPDRO4XNCXDN3QFZNGI5NAJA5DSYHXKRUXQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.727396258 (33m19.50832964s) [noise=true , address=104.36.82.55:4228 , piece_id=FGXP2NNTU5IATOQKF6FHPXTO4A4NA3N7BCIFWM3IQESGGSF64UBQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:04.363269378 (33m20.144202762s) [noise=true , address=64.49.13.140:4508 , piece_id=7H3YCEUT26RWXLJUV5EQSQRG6SBUX4K5PI3THT7FPYUQXGEZS3CA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:05.662641144 (33m21.443574531s) [noise=true , address=64.49.13.169:4716 , piece_id=CUIDRZDYM3XWO4UOTF3YDHEKPMBTAOTNG3OV63BKUXSMOBEUBQNQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:26:03.141300303 (34m18.922233688s) [noise=true , address=104.36.82.40:4953 , piece_id=755HXZWUZXT4RKFQF3NSSJPTLKKQWO4O2NBRU6LPTKJU5MNAET6A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: rpc: dial tcp 104.36.82.40:4953: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141414422 (34m18.9223478s) [noise=true , address=104.36.82.56:4234 , piece_id=OPYBCBF4AJPAUHMP5UCO4WBM7RJ2RH7ZEHOGAWTDSPGKFKXVLAZA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: write tcp 159.223.143.39:48556->104.36.82.56:4234: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141479294 (34m18.922412673s) [noise=true , address=104.36.82.2:4755 , piece_id=V4EX5DHU2FBKS6PBGHDKVVVWLX2IPHFE3PDVTBLEWU2ANQQ5K27A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: rpc: dial tcp 104.36.82.2:4755: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141548226 (34m18.922481603s) [noise=true , address=77.90.31.17:4796 , piece_id=PIDEYKX37B4DDG3S2W5JCUOGXGUSZF57RCADSY3FESNJVIL4MVPA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 159.223.143.39:32880->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141618263 (34m18.922551641s) [noise=true , address=104.36.82.5:4854 , piece_id=BTT3UUHTR2JCQ7473Y4ZWGS6JAF7VQFADW2HKOBBR4EOJWY5JV5A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141682694 (34m18.922616074s) [noise=true , address=104.36.82.11:4146 , piece_id=TXPTK6Q4Y5ZE4IVEDMNXM4YTTB6CIA3VN7ZUFQQALIE5MIIEP6KA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1Sv7eCy62QTiVT1c7mxzJSTAnrCsDoRrxqxfL6govEFRJRQJ6V, address:104.36.82.11:4146): protocol: rpc: dial tcp 104.36.82.11:4146: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141742715 (34m18.922676102s) [noise=true , address=104.36.82.42:4817 , piece_id=EVFDOVL3OIPCCXAGBCMHVLNZ2IRIRV7JLGZESNXMJ3LUVKRPDSTQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: rpc: dial tcp 104.36.82.42:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141807146 (34m18.922740633s) [noise=true , address=104.36.82.13:4199 , piece_id=2WM67WHZZLFCUU42RYWARGL64IMLNXNL7RNXE7CA75HPCFVLOD6A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 159.223.143.39:32852->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141873320 (34m18.922806698s) [noise=true , address=104.36.82.7:4922 , piece_id=5MLUACJA4KYMIHUHL5M4RF2WSEGORQCPDQTC4X46NZ2QVYQBLK7A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: write tcp 159.223.143.39:40742->104.36.82.7:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141935463 (34m18.922868845s) [noise=true , address=104.36.82.51:4084 , piece_id=UEE5PVLTDFFAIPEBUZXP5VUILHX5FSNICCREOTDU4MWJIX74JUJA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9, address:104.36.82.51:4084): protocol: write tcp 159.223.143.39:44906->104.36.82.51:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141956884 (34m18.922890269s) [seg_pos={0 0}, upload=522]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:26:03.169669305 (34m18.950602669s) [upload=522]: Done uploading segments... Jul 26 22:26:03.169699803 (34m18.95063317s) [upload=522]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.033538616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [............X..X........XXX.XXX.X.XX..X..X......X...........] duration: 570.779913ms FAIL (2025-07-26 22:26:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 570.806768ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 593.92687ms FAIL (2025-07-26 22:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.956464ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................XX..XXX..X..XX..X..XX.X....X....X........X] duration: 30.030608824s FAIL (2025-07-26 22:25:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.030649121s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)