Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXX] duration: 1m0.031443156s FAIL (2025-07-26 23:11:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== id=R4GRXYBCXYPNRW7JKKOMPDAWBJVJEJIXCSI7SQZIJQPPOJVXBHXQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.870187335 (1h18m57.651120805s) [noise=true , address=45.91.116.9:4267 , piece_id=ZJW5Z2LQRC4UGLNDUMHAENNCMOYLKPMAHBK3WD4PVNE4SVYP4OIA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.870616409 (1h18m57.651549774s) [noise=true , address=79.127.219.38:4005 , piece_id=RUHAVD6QHNAZTFALXZ5R2LGCHUPXON643XTSYH35WUFN5JTBIYKA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.870995275 (1h18m57.651928659s) [noise=true , address=104.36.82.52:4117 , piece_id=FUK4U7MTII5KU3A5X57PQ2FGXJKBH2TPYZG7JWOPU2BH5M7YAFWA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.871184821 (1h18m57.652118204s) [noise=true , address=45.91.116.5:4310 , piece_id=EDYRH4QTQEZTCBPB5MOJDHQLP733UOYM2SQN3UIYMCEXSLQQPDEA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.871752249 (1h18m57.652685637s) [noise=true , address=79.127.205.234:4613 , piece_id=JXBLE7PWJPDC2H7AJCYJ54KU5HS5F3724GNRP7GFCVJKJN44TIVA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.872541010 (1h18m57.653474396s) [noise=true , address=64.49.13.140:4508 , piece_id=JYAPTS7LWWDZVU56IUMH2CL36RN4EREX6LGZ7HYZWKC2AVJB7DVQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.872941914 (1h18m57.653875303s) [noise=true , address=104.36.82.54:4168 , piece_id=2LQLSPF2WBEB5NPYHXFXK6SFLNW7MZPUTW73LWNIMGHSRR3TRBOQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.873405451 (1h18m57.654338833s) [noise=true , address=79.127.205.245:4239 , piece_id=FGODP7DP4J4EKSMLBSDDXBR6SMR4SBA5V3LCCJVVXRLEMEU6N7FA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.873832557 (1h18m57.654765946s) [noise=true , address=104.36.82.53:4141 , piece_id=Z65GGMKN3PXYYP5Y4EHD2UHVUKR7YCW73PPKT5DUJEJEKYW3VJPQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.874322471 (1h18m57.655256059s) [noise=true , address=77.90.31.15:4731 , piece_id=GQ3Z32JRBSWLCBJKJKYSNREB6BTLB5ZIQUXHFF43O6GBRLBEVMJA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.874885264 (1h18m57.655818643s) [noise=true , address=79.127.220.97:4361 , piece_id=7OBQSMYEA2SQZ4NGHO2Q3D7F7M5VTVHORDEPWS6EZKPGVLMVYECA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.875129299 (1h18m57.656062684s) [noise=true , address=104.36.82.47:4524 , piece_id=33OXGVKYVQYD3IDKCNPD76CPJL5RTPJAQPW6ACBJDZTG2XT2NPPA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.875556485 (1h18m57.656489874s) [noise=true , address=64.49.13.155:4916 , piece_id=RMAGQMUWYXPXIIJQRIYV5MQOKFKQEKDSCVEAUZQ6IMSSY5KBCWVQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.875974647 (1h18m57.65690803s) [noise=true , address=77.90.31.11:4557 , piece_id=CNAXJYRA6XDTLZY3CDJZGN535A7END3Z24HSBOWTEJRUZFUIB6EQ, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.876491401 (1h18m57.657424796s) [noise=true , address=104.36.82.55:4198 , piece_id=XQU6BLGD2YOZAHK5SAFWE34QNAIXPXFQDYWEPQR76KOKGFVEJFFA, seg_pos={0 0}, upload=1214]: Uploading piece... Jul 26 23:10:41.946007955 (1h18m57.726941332s) [upload=1214]: Next returned nil segment Jul 26 23:10:41.946028810 (1h18m57.726962173s) [upload=1214]: Waiting for error group managing segments... Jul 26 23:10:41.974997422 (1h18m57.755930783s) [noise=true , address=79.127.219.38:4005 , piece_id=RUHAVD6QHNAZTFALXZ5R2LGCHUPXON643XTSYH35WUFN5JTBIYKA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:41.975066158 (1h18m57.755999522s) [noise=true , address=77.90.31.11:4557 , piece_id=CNAXJYRA6XDTLZY3CDJZGN535A7END3Z24HSBOWTEJRUZFUIB6EQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:42.191500436 (1h18m57.972433836s) [noise=true , address=79.127.220.97:4361 , piece_id=7OBQSMYEA2SQZ4NGHO2Q3D7F7M5VTVHORDEPWS6EZKPGVLMVYECA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:42.429147699 (1h18m58.210081079s) [noise=true , address=79.127.205.245:4239 , piece_id=FGODP7DP4J4EKSMLBSDDXBR6SMR4SBA5V3LCCJVVXRLEMEU6N7FA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:42.486215576 (1h18m58.267148954s) [noise=true , address=79.127.205.234:4613 , piece_id=JXBLE7PWJPDC2H7AJCYJ54KU5HS5F3724GNRP7GFCVJKJN44TIVA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:42.733589903 (1h18m58.514523293s) [noise=true , address=77.90.31.15:4731 , piece_id=GQ3Z32JRBSWLCBJKJKYSNREB6BTLB5ZIQUXHFF43O6GBRLBEVMJA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:10:43.771046722 (1h18m59.551980102s) [noise=true , address=45.91.116.9:4267 , piece_id=ZJW5Z2LQRC4UGLNDUMHAENNCMOYLKPMAHBK3WD4PVNE4SVYP4OIA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:10:44.125640277 (1h18m59.906573664s) [noise=true , address=64.49.13.140:4508 , piece_id=JYAPTS7LWWDZVU56IUMH2CL36RN4EREX6LGZ7HYZWKC2AVJB7DVQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:10:45.351934876 (1h19m1.132868253s) [noise=true , address=45.91.116.5:4310 , piece_id=EDYRH4QTQEZTCBPB5MOJDHQLP733UOYM2SQN3UIYMCEXSLQQPDEA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 23:10:46.147524007 (1h19m1.92845737s) [noise=true , address=64.49.13.155:4916 , piece_id=RMAGQMUWYXPXIIJQRIYV5MQOKFKQEKDSCVEAUZQ6IMSSY5KBCWVQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: <nil> Jul 26 23:11:41.798905663 (1h19m57.579839045s) [noise=true , address=104.36.82.46:4482 , piece_id=R4GRXYBCXYPNRW7JKKOMPDAWBJVJEJIXCSI7SQZIJQPPOJVXBHXQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.798991109 (1h19m57.579924489s) [noise=true , address=104.36.82.51:4063 , piece_id=GBPD63H4PLAFY56M7ZDQ62EJY2V4W4FLI3HOHW545Y5ZZZPQTVFA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D, address:104.36.82.51:4063): protocol: rpc: dial tcp 104.36.82.51:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799120082 (1h19m57.580053461s) [noise=true , address=77.90.31.12:4608 , piece_id=6Z6I6OQOZXKE2LBFCTOJKO3SUQZ424JEPEARBDFJO4YTEPZTXZGQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799201382 (1h19m57.580134748s) [noise=true , address=104.36.82.54:4168 , piece_id=2LQLSPF2WBEB5NPYHXFXK6SFLNW7MZPUTW73LWNIMGHSRR3TRBOQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: write tcp 159.223.143.39:56912->104.36.82.54:4168: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799229015 (1h19m57.58016243s) [noise=true , address=104.36.82.55:4198 , piece_id=XQU6BLGD2YOZAHK5SAFWE34QNAIXPXFQDYWEPQR76KOKGFVEJFFA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:35482->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799263143 (1h19m57.580196519s) [noise=true , address=104.36.82.53:4141 , piece_id=Z65GGMKN3PXYYP5Y4EHD2UHVUKR7YCW73PPKT5DUJEJEKYW3VJPQ, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799289517 (1h19m57.580222896s) [noise=true , address=104.36.82.52:4117 , piece_id=FUK4U7MTII5KU3A5X57PQ2FGXJKBH2TPYZG7JWOPU2BH5M7YAFWA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: rpc: dial tcp 104.36.82.52:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799330906 (1h19m57.580264273s) [noise=true , address=104.36.82.47:4524 , piece_id=33OXGVKYVQYD3IDKCNPD76CPJL5RTPJAQPW6ACBJDZTG2XT2NPPA, seg_pos={0 0}, upload=1214]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:36736->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:41.799358985 (1h19m57.580292371s) [seg_pos={0 0}, upload=1214]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:11:41.827662811 (1h19m57.608596179s) [upload=1214]: Done uploading segments... Jul 26 23:11:41.827709125 (1h19m57.608642504s) [upload=1214]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.03146163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 385.35201ms FAIL (2025-07-26 23:11:06 UTC) orbiter warning: uplink: too many requests Duration: 385.405564ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX.X.] duration: 9.5002635s FAIL (2025-07-26 23:11:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.50029717s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...................................................XXXXX] duration: 30.032589847s FAIL (2025-07-26 23:10:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== em_ObjectBegin Jul 26 23:10:28.254218390 (1h18m44.035151769s) [upload=1210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:10:28.285287665 (1h18m44.066221076s) [seg_pos={0 0}, upload=1210]: Begin upload segment... Jul 26 23:10:28.285350233 (1h18m44.066283615s) [seg_pos={0 0}, upload=1210]: Done begin upload segment. Jul 26 23:10:28.285386097 (1h18m44.066319486s) [noise=true , address=45.91.116.12:4226 , piece_id=LFCQAAR5JAQIRZHFT53RPHDNZIVAASBJ3ISPA74QNSYQR4Y6ISLA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.285946221 (1h18m44.06687961s) [noise=true , address=104.36.82.2:4729 , piece_id=WX23WUWJ4TTUY46DGMXY2LH4MGKNCGEBAZBZEZM3YEPGOXEUA3RA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.286406564 (1h18m44.067339955s) [noise=true , address=77.90.31.11:4553 , piece_id=DUN7GZAOBEUISYTM7CG3XMLR4TTD3KA7AP53UIPQCCKGONN7STSA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.286844038 (1h18m44.067777425s) [noise=true , address=45.91.116.7:4011 , piece_id=FO5SPBT7T5TCL4NO5UNCHINO7ORVHIGOAAWVR5UZVHFV7KJ2722A, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.287374383 (1h18m44.068307769s) [noise=true , address=104.36.82.3:4765 , piece_id=BJXKL5PBV6ODWHLG7VKTY7PCBJAMPHLWV4VDX4UPXVP4OVE2LTXA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.288249677 (1h18m44.069183245s) [noise=true , address=104.36.82.6:4875 , piece_id=47YVS2SIO57X6BQK7U6SRZHOGXWK2OC6B5L53HX4Q5VVRDZBUAWQ, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.289011232 (1h18m44.06994462s) [noise=true , address=104.36.82.15:4257 , piece_id=DDHGTXUZNIS7HVDOUFY2XWYCHK5K4TSMGPDHYTI3BDFRFUUSCMBQ, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.289744156 (1h18m44.070677541s) [noise=true , address=77.90.31.14:4663 , piece_id=6OBPGU47AGZLKP6WAZN6D2SVYBEXKVTGPMFOBY4UBD6CAY5QPQ4Q, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.290309187 (1h18m44.071242577s) [noise=true , address=104.36.82.11:4115 , piece_id=66O5GCZ6LNIXVFW4XI674I7IDKA6OS42VPYE57VB3C2WR264MAOA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.291212313 (1h18m44.072145705s) [noise=true , address=104.36.82.4:4801 , piece_id=MTJLKYMFITXBD46BUGZI4L4FFVSEF2ELDV6CZ3PTYOUJ7VOGTX2Q, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.291748520 (1h18m44.072681909s) [noise=true , address=45.91.116.6:4975 , piece_id=V25T7FDODQRCXDNJ3O6NAY6DJR2X4NLLKTQHYMOZF7NMUEI3O45Q, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.292331044 (1h18m44.073264434s) [noise=true , address=104.36.82.9:4005 , piece_id=ETMQVPV753VSG7RJVX3BDO6EAQPIDZRY4K575MU2VSZ6SJ55DHJA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.292822957 (1h18m44.073756345s) [noise=true , address=104.36.82.16:4293 , piece_id=ZU75IIPUV6N6MRIKS2WUHWQMMGFBQUPZVSC4CRBTMATYAJQHUPJA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.293275001 (1h18m44.074208381s) [noise=true , address=104.36.82.8:4972 , piece_id=VU3T2PB5K2C35FAMJD6OSNXRVRX3SWHHJEZX5LB5FYYIS3GRHSBQ, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.293607206 (1h18m44.074540597s) [noise=true , address=77.90.31.13:4627 , piece_id=UZOOWWIYR32SRTK2VB5LOM4QIF2QJSFM7XLF5646KZC3MPCQ6KDA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.294008911 (1h18m44.074942299s) [noise=true , address=45.91.116.8:4048 , piece_id=WHRT6Z3V4VCGYC7BHIABKOPQ3V3IASY55CGUXW64ODJF6VHM6NOA, seg_pos={0 0}, upload=1210]: Uploading piece... Jul 26 23:10:28.382224573 (1h18m44.163157944s) [upload=1210]: Next returned nil segment Jul 26 23:10:28.382242699 (1h18m44.163176084s) [upload=1210]: Waiting for error group managing segments... Jul 26 23:10:28.389256579 (1h18m44.170189959s) [noise=true , address=45.91.116.8:4048 , piece_id=WHRT6Z3V4VCGYC7BHIABKOPQ3V3IASY55CGUXW64ODJF6VHM6NOA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:28.390139836 (1h18m44.171073205s) [noise=true , address=45.91.116.12:4226 , piece_id=LFCQAAR5JAQIRZHFT53RPHDNZIVAASBJ3ISPA74QNSYQR4Y6ISLA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:28.390754156 (1h18m44.171687527s) [noise=true , address=45.91.116.6:4975 , piece_id=V25T7FDODQRCXDNJ3O6NAY6DJR2X4NLLKTQHYMOZF7NMUEI3O45Q, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:28.390849392 (1h18m44.171782767s) [noise=true , address=45.91.116.7:4011 , piece_id=FO5SPBT7T5TCL4NO5UNCHINO7ORVHIGOAAWVR5UZVHFV7KJ2722A, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:28.391346651 (1h18m44.172280018s) [noise=true , address=77.90.31.14:4663 , piece_id=6OBPGU47AGZLKP6WAZN6D2SVYBEXKVTGPMFOBY4UBD6CAY5QPQ4Q, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:28.400244833 (1h18m44.181178213s) [noise=true , address=77.90.31.11:4553 , piece_id=DUN7GZAOBEUISYTM7CG3XMLR4TTD3KA7AP53UIPQCCKGONN7STSA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: <nil> Jul 26 23:10:58.214276461 (1h19m13.995209831s) [noise=true , address=104.36.82.6:4875 , piece_id=47YVS2SIO57X6BQK7U6SRZHOGXWK2OC6B5L53HX4Q5VVRDZBUAWQ, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214363793 (1h19m13.995297164s) [noise=true , address=104.36.82.11:4115 , piece_id=66O5GCZ6LNIXVFW4XI674I7IDKA6OS42VPYE57VB3C2WR264MAOA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214632880 (1h19m13.995566249s) [noise=true , address=104.36.82.2:4729 , piece_id=WX23WUWJ4TTUY46DGMXY2LH4MGKNCGEBAZBZEZM3YEPGOXEUA3RA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51852->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214713410 (1h19m13.995646771s) [noise=true , address=77.90.31.13:4627 , piece_id=UZOOWWIYR32SRTK2VB5LOM4QIF2QJSFM7XLF5646KZC3MPCQ6KDA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 159.223.143.39:40982->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214730247 (1h19m13.995663631s) [noise=true , address=104.36.82.15:4257 , piece_id=DDHGTXUZNIS7HVDOUFY2XWYCHK5K4TSMGPDHYTI3BDFRFUUSCMBQ, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214800817 (1h19m13.995734192s) [noise=true , address=104.36.82.4:4801 , piece_id=MTJLKYMFITXBD46BUGZI4L4FFVSEF2ELDV6CZ3PTYOUJ7VOGTX2Q, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214832000 (1h19m13.995765378s) [noise=true , address=104.36.82.8:4972 , piece_id=VU3T2PB5K2C35FAMJD6OSNXRVRX3SWHHJEZX5LB5FYYIS3GRHSBQ, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214889384 (1h19m13.995822761s) [noise=true , address=104.36.82.9:4005 , piece_id=ETMQVPV753VSG7RJVX3BDO6EAQPIDZRY4K575MU2VSZ6SJ55DHJA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214905765 (1h19m13.995839142s) [noise=true , address=104.36.82.3:4765 , piece_id=BJXKL5PBV6ODWHLG7VKTY7PCBJAMPHLWV4VDX4UPXVP4OVE2LTXA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:40000->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.214955954 (1h19m13.995889323s) [noise=true , address=104.36.82.16:4293 , piece_id=ZU75IIPUV6N6MRIKS2WUHWQMMGFBQUPZVSC4CRBTMATYAJQHUPJA, seg_pos={0 0}, upload=1210]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:58216->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:58.215057029 (1h19m13.995990405s) [seg_pos={0 0}, upload=1210]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:10:58.244821394 (1h19m14.025754772s) [upload=1210]: Done uploading segments... Jul 26 23:10:58.244872146 (1h19m14.025805526s) [upload=1210]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.032642002s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.X.X...XX.................X.X..........XX...X...X..XX.X...] duration: 28.455255931s FAIL (2025-07-26 23:10:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.455283803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXX.X] duration: 30.001748872s FAIL (2025-07-26 23:10:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001776846s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X................X........................X......XXXXX] duration: 30.053852388s FAIL (2025-07-26 23:10:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.053873031s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 246.046267ms FAIL (2025-07-26 23:10:48 UTC) orbiter warning: uplink: too many requests Duration: 246.081261ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXX] duration: 1m0.030019782s FAIL (2025-07-26 23:10:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== n upload segment... Jul 26 23:09:41.800910326 (1h17m57.581843704s) [seg_pos={0 0}, upload=1198]: Done begin upload segment. Jul 26 23:09:41.800946276 (1h17m57.581879665s) [noise=true , address=104.36.82.60:4361 , piece_id=LRBHSO5KDLNHIEDUNOQUTVSPUOCR4ARXYSYI7YV4JWYOY62GQ2NA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.801457785 (1h17m57.582391173s) [noise=true , address=104.36.82.46:4491 , piece_id=LAWMWH2SSO4VQ7UZ43ENUWHM4BZXUXCNGQMG526BTLI5HB7TWWOQ, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.801838466 (1h17m57.582771853s) [noise=true , address=79.127.219.48:4090 , piece_id=THAW2XVB57GHMGAAV5WYEULCUILR4KE6DAAULLEEAUBMAW4KIKCQ, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.802046517 (1h17m57.582979906s) [noise=true , address=45.91.116.5:4310 , piece_id=SP3Z7AJSBZAPURHTW6WMIXKMORVH4HYAY63CD6WPL5EO7OZT6XKA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.802252626 (1h17m57.583186011s) [noise=true , address=79.127.205.233:4652 , piece_id=T7CO5XLXBA2M7V6ZXDWBTXGNYCK2VWVX5VLBTDUYG3UIE4JWB2AA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.802680482 (1h17m57.583613862s) [noise=true , address=64.49.13.155:4916 , piece_id=PAY6KVVZDFYPICB63ATG5MUQQQJLAH57AMCLWA5WLHWSXECKZMJA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.802722117 (1h17m57.583655503s) [noise=true , address=104.36.82.55:4197 , piece_id=QBPLHM6MQP4AU2AUUIDI27HNSUBBJEN55VJ5YRDD7HJLO6JLU66Q, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.803337999 (1h17m57.584271388s) [noise=true , address=79.127.205.245:4243 , piece_id=RLHEXWMW34BWQKWVIS66GBYWESVB2MNSQCTFA6ECFJY33CAELFTA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.807709772 (1h17m57.588643158s) [noise=true , address=156.146.43.225:4449 , piece_id=HWLNI4LWUNAAPSLWEXAMLREDZ7XENGDCOXBBYH7JD4JSQMVQSU4A, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.808620294 (1h17m57.589553683s) [noise=true , address=79.127.201.215:4346 , piece_id=QMRDJ2VNDMOFH2QJNYKVZKLWD4XECNW463FEQT5LM6VASYTWEOMA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.809169072 (1h17m57.59010246s) [noise=true , address=104.36.82.47:4532 , piece_id=VDVZ3DJTKNALTPF2FOVGB5RMUSJ34FL6G5QUE25SGUIKR4T7P6HQ, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.809572342 (1h17m57.590505736s) [noise=true , address=104.36.82.54:4172 , piece_id=VPNPBKWKXQRTDTUX4QJ7F7WYJDXTRZAB7KOCBAXRUIIRCXQGFRRQ, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.809967384 (1h17m57.590900771s) [noise=true , address=45.91.116.14:4194 , piece_id=GWIAK2SC45ITZKP3GQZFSD5YNBPZA7XSIQHOUMNC2ZXSOPZC2BTA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.810429537 (1h17m57.591362923s) [noise=true , address=104.36.82.52:4096 , piece_id=AACZDDT3STYV7QV4XYRRRTBMYH6JWKY5ZFXRH2PBYQ4WT6SORLCA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.810833189 (1h17m57.591766576s) [noise=true , address=79.127.219.47:4048 , piece_id=P67HEPJOWKU77QYRLOEGEUZBGZVEH6DUOLSZWSECRUBPJ7JKLKYA, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.811287429 (1h17m57.592220816s) [noise=true , address=77.90.31.18:4821 , piece_id=H6CYBQMM7VKK4XB5TXGR6MHJUVNDOEYRWEQZNSJNGI5D6HACX56Q, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.811646541 (1h17m57.592579925s) [noise=true , address=77.90.31.11:4581 , piece_id=GSFMO7YN64TP32MCFQRLCNZXD4YYBRHRO4ISKQRNUJT3U4HKT35Q, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.812011067 (1h17m57.592944454s) [noise=true , address=64.49.13.140:4508 , piece_id=ZRSZC5IIFJVHKPW2UN2M5OWWJSMGF3Q4TTCMSGOD7L2R7J2Y2L4Q, seg_pos={0 0}, upload=1198]: Uploading piece... Jul 26 23:09:41.885912023 (1h17m57.666845391s) [upload=1198]: Next returned nil segment Jul 26 23:09:41.888846083 (1h17m57.669779468s) [upload=1198]: Waiting for error group managing segments... Jul 26 23:09:41.938500099 (1h17m57.719433479s) [noise=true , address=77.90.31.11:4581 , piece_id=GSFMO7YN64TP32MCFQRLCNZXD4YYBRHRO4ISKQRNUJT3U4HKT35Q, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:41.995392021 (1h17m57.7763254s) [noise=true , address=77.90.31.18:4821 , piece_id=H6CYBQMM7VKK4XB5TXGR6MHJUVNDOEYRWEQZNSJNGI5D6HACX56Q, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:42.117440437 (1h17m57.898373799s) [noise=true , address=79.127.201.215:4346 , piece_id=QMRDJ2VNDMOFH2QJNYKVZKLWD4XECNW463FEQT5LM6VASYTWEOMA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:42.160233902 (1h17m57.941167283s) [noise=true , address=104.36.82.60:4361 , piece_id=LRBHSO5KDLNHIEDUNOQUTVSPUOCR4ARXYSYI7YV4JWYOY62GQ2NA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:42.163633347 (1h17m57.94456673s) [noise=true , address=156.146.43.225:4449 , piece_id=HWLNI4LWUNAAPSLWEXAMLREDZ7XENGDCOXBBYH7JD4JSQMVQSU4A, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:42.299441102 (1h17m58.080374478s) [noise=true , address=45.91.116.14:4194 , piece_id=GWIAK2SC45ITZKP3GQZFSD5YNBPZA7XSIQHOUMNC2ZXSOPZC2BTA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:42.410062498 (1h17m58.190995879s) [noise=true , address=79.127.205.245:4243 , piece_id=RLHEXWMW34BWQKWVIS66GBYWESVB2MNSQCTFA6ECFJY33CAELFTA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:42.458585588 (1h17m58.239518955s) [noise=true , address=64.49.13.155:4916 , piece_id=PAY6KVVZDFYPICB63ATG5MUQQQJLAH57AMCLWA5WLHWSXECKZMJA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:42.543605830 (1h17m58.324539208s) [noise=true , address=79.127.205.233:4652 , piece_id=T7CO5XLXBA2M7V6ZXDWBTXGNYCK2VWVX5VLBTDUYG3UIE4JWB2AA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:43.788745108 (1h17m59.569678477s) [noise=true , address=79.127.219.48:4090 , piece_id=THAW2XVB57GHMGAAV5WYEULCUILR4KE6DAAULLEEAUBMAW4KIKCQ, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:44.334064562 (1h18m0.114997931s) [noise=true , address=79.127.219.47:4048 , piece_id=P67HEPJOWKU77QYRLOEGEUZBGZVEH6DUOLSZWSECRUBPJ7JKLKYA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:49.607797841 (1h18m5.388731208s) [noise=true , address=64.49.13.140:4508 , piece_id=ZRSZC5IIFJVHKPW2UN2M5OWWJSMGF3Q4TTCMSGOD7L2R7J2Y2L4Q, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:09:50.481477880 (1h18m6.262411252s) [noise=true , address=45.91.116.5:4310 , piece_id=SP3Z7AJSBZAPURHTW6WMIXKMORVH4HYAY63CD6WPL5EO7OZT6XKA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: <nil> Jul 26 23:10:41.704560996 (1h18m57.485494383s) [noise=true , address=104.36.82.55:4197 , piece_id=QBPLHM6MQP4AU2AUUIDI27HNSUBBJEN55VJ5YRDD7HJLO6JLU66Q, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 159.223.143.39:36916->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:41.704630520 (1h18m57.485563897s) [noise=true , address=104.36.82.47:4532 , piece_id=VDVZ3DJTKNALTPF2FOVGB5RMUSJ34FL6G5QUE25SGUIKR4T7P6HQ, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 159.223.143.39:38246->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:41.704649972 (1h18m57.485583349s) [noise=true , address=104.36.82.46:4491 , piece_id=LAWMWH2SSO4VQ7UZ43ENUWHM4BZXUXCNGQMG526BTLI5HB7TWWOQ, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: write tcp 159.223.143.39:47654->104.36.82.46:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:41.704687029 (1h18m57.485620399s) [noise=true , address=104.36.82.54:4172 , piece_id=VPNPBKWKXQRTDTUX4QJ7F7WYJDXTRZAB7KOCBAXRUIIRCXQGFRRQ, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 159.223.143.39:45308->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:41.704716181 (1h18m57.485649558s) [noise=true , address=104.36.82.52:4096 , piece_id=AACZDDT3STYV7QV4XYRRRTBMYH6JWKY5ZFXRH2PBYQ4WT6SORLCA, seg_pos={0 0}, upload=1198]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:41.704746784 (1h18m57.485680163s) [seg_pos={0 0}, upload=1198]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:10:41.731634190 (1h18m57.51256756s) [upload=1198]: Done uploading segments... Jul 26 23:10:41.731658054 (1h18m57.51259142s) [upload=1198]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.030041549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.X.X...XX.................X.X..........XX...X...X..XX.X..] duration: 32.631315855s FAIL (2025-07-26 23:10:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.631344315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...................................................XXXX] duration: 30.046133342s FAIL (2025-07-26 23:10:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== sdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:09:48.394614130 (1h18m4.175547518s) [seg_pos={0 0}, upload=1193]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:09:48.421850894 (1h18m4.20278427s) [upload=1193]: Done uploading segments... Jul 26 23:09:48.421882670 (1h18m4.202816041s) [upload=1193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:09:48.527130539 (1h18m4.308063924s) [upload=1204]: Starting upload Jul 26 23:09:48.527210849 (1h18m4.308144213s) [upload=1204]: Done starting upload Jul 26 23:09:48.527555475 (1h18m4.308488843s) [upload=1204]: Uploading segments... Jul 26 23:09:48.527611766 (1h18m4.308545155s) [upload=1204]: Got next segment. Inline: false Jul 26 23:09:48.527652951 (1h18m4.308586316s) [upload=1204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:09:48.527659524 (1h18m4.308592914s) [upload=1204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:09:48.563835118 (1h18m4.344768507s) [seg_pos={0 0}, upload=1204]: Begin upload segment... Jul 26 23:09:48.563891103 (1h18m4.344824487s) [seg_pos={0 0}, upload=1204]: Done begin upload segment. Jul 26 23:09:48.563921300 (1h18m4.344854686s) [noise=true , address=77.90.31.11:4554 , piece_id=G5THYZ4YECGUHTCPALAH3IXUXECCUHMPI4DHJRQYFX5FG6T2U23A, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.564571702 (1h18m4.345505087s) [noise=true , address=104.36.82.15:4259 , piece_id=SKYAST2XA2N6BUYHVJSJZCGLMVHY26K3IKMMJJPRVUKNDZJS75GQ, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.564962998 (1h18m4.345896386s) [noise=true , address=104.36.82.9:4006 , piece_id=YID4ZLFSY6MVECT6KIPOEISCCZTWKJPRFJUQ3HOLQZP4VFWBYO7A, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.565374235 (1h18m4.346307623s) [noise=true , address=45.91.116.8:4048 , piece_id=RNNRYCQBXM5DL2NWAFOBPPQT3WIJSMGLCCQ4BFYUC4S2F3RQK2DA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.565802432 (1h18m4.34673582s) [noise=true , address=104.36.82.8:4971 , piece_id=HOLAVC7VFC7M6OG37S5DU5YPAUZECGK3DZZ4GD2JFQNCGZX4FYPQ, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.566238110 (1h18m4.347171499s) [noise=true , address=104.36.82.14:4222 , piece_id=H4JDYLGC5JU2FIJE6YYY2JDWOPOHKCPLPPOZORR5ML5RWYVTTUYQ, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.569017454 (1h18m4.349950841s) [noise=true , address=77.90.31.19:4843 , piece_id=27A276W72M3XKOLY2XYZVTMMWYMPBSUG5G54VE6YNRVTN5X3QLLA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.569494281 (1h18m4.350427673s) [noise=true , address=45.91.116.10:4083 , piece_id=TBJETQVA62WA2COEJP6TL5XD44K6Z2XZFRKIMY4VGELSPCRVYXVA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.570708651 (1h18m4.351642038s) [noise=true , address=45.91.116.4:4938 , piece_id=CPDNLGILRUC67MAA7X2XA22N2SW7FPOUE3VXJZI6ZPA75N2INE7A, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.579451876 (1h18m4.360385265s) [noise=true , address=104.36.82.6:4875 , piece_id=QVPT4PSLM43FFX2UMDMV7LHD4NBLTTHV6RXNM5NLSTEXOXR47AQA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.580091827 (1h18m4.361025194s) [noise=true , address=104.36.82.14:4223 , piece_id=RPJB2O24KCERXUNZ4DJQEI3JFEAWQDLIN3U3IVW57QIYGP5MQFWA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.580524144 (1h18m4.361457528s) [noise=true , address=104.36.82.16:4294 , piece_id=HLYZFPCCGUP3RSECY4NMMY6JDO6FQS3RS33CKGSLFOCXLSDKP6UA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.580930198 (1h18m4.361863579s) [noise=true , address=45.91.116.6:4975 , piece_id=NOMEXDCYEZS3GGHKZDBYSJCD7YPFJEPMW2QA72CUOZWSSV5C7E5Q, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.581367328 (1h18m4.362300813s) [noise=true , address=45.91.116.14:4191 , piece_id=36YEELH55BDYXC7VQAZ53RAU66HVKYU5NVSYGWQIPABALYZYACEQ, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.581957817 (1h18m4.362891207s) [noise=true , address=104.36.82.6:4874 , piece_id=PZY35ASQBO7MRVGJF6OMON3AHHEXRXSPRUTBFXZFI5LIBVLJNSUQ, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.582481701 (1h18m4.363415085s) [noise=true , address=45.91.116.7:4011 , piece_id=YRWZEVTAFSSIBYZEGGNRTLS35UVHP47SS7AWMW2EGBRIJK24ILGA, seg_pos={0 0}, upload=1204]: Uploading piece... Jul 26 23:09:48.654206791 (1h18m4.435140155s) [upload=1204]: Next returned nil segment Jul 26 23:09:48.654225420 (1h18m4.4351588s) [upload=1204]: Waiting for error group managing segments... Jul 26 23:09:48.659509836 (1h18m4.440443206s) [noise=true , address=45.91.116.14:4191 , piece_id=36YEELH55BDYXC7VQAZ53RAU66HVKYU5NVSYGWQIPABALYZYACEQ, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.660140872 (1h18m4.441074236s) [noise=true , address=45.91.116.7:4011 , piece_id=YRWZEVTAFSSIBYZEGGNRTLS35UVHP47SS7AWMW2EGBRIJK24ILGA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.660293658 (1h18m4.441227024s) [noise=true , address=45.91.116.4:4938 , piece_id=CPDNLGILRUC67MAA7X2XA22N2SW7FPOUE3VXJZI6ZPA75N2INE7A, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.660834542 (1h18m4.441767912s) [noise=true , address=45.91.116.10:4083 , piece_id=TBJETQVA62WA2COEJP6TL5XD44K6Z2XZFRKIMY4VGELSPCRVYXVA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.661197689 (1h18m4.442131063s) [noise=true , address=45.91.116.6:4975 , piece_id=NOMEXDCYEZS3GGHKZDBYSJCD7YPFJEPMW2QA72CUOZWSSV5C7E5Q, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.661279755 (1h18m4.442213121s) [noise=true , address=45.91.116.8:4048 , piece_id=RNNRYCQBXM5DL2NWAFOBPPQT3WIJSMGLCCQ4BFYUC4S2F3RQK2DA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.673569874 (1h18m4.454503253s) [noise=true , address=77.90.31.11:4554 , piece_id=G5THYZ4YECGUHTCPALAH3IXUXECCUHMPI4DHJRQYFX5FG6T2U23A, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:09:48.684142941 (1h18m4.465076317s) [noise=true , address=77.90.31.19:4843 , piece_id=27A276W72M3XKOLY2XYZVTMMWYMPBSUG5G54VE6YNRVTN5X3QLLA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: <nil> Jul 26 23:10:18.489679219 (1h18m34.270612606s) [noise=true , address=104.36.82.14:4222 , piece_id=H4JDYLGC5JU2FIJE6YYY2JDWOPOHKCPLPPOZORR5ML5RWYVTTUYQ, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:18.489782269 (1h18m34.27071565s) [noise=true , address=104.36.82.9:4006 , piece_id=YID4ZLFSY6MVECT6KIPOEISCCZTWKJPRFJUQ3HOLQZP4VFWBYO7A, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:18.489839214 (1h18m34.270772593s) [noise=true , address=104.36.82.8:4971 , piece_id=HOLAVC7VFC7M6OG37S5DU5YPAUZECGK3DZZ4GD2JFQNCGZX4FYPQ, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:18.489936434 (1h18m34.270869814s) [noise=true , address=104.36.82.6:4875 , piece_id=QVPT4PSLM43FFX2UMDMV7LHD4NBLTTHV6RXNM5NLSTEXOXR47AQA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:10:18.490015809 (1h18m34.27094919s) [noise=true , address=104.36.82.15:4259 , piece_id=SKYAST2XA2N6BUYHVJSJZCGLMVHY26K3IKMMJJPRVUKNDZJS75GQ, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:10:18.490152890 (1h18m34.271086282s) [noise=true , address=104.36.82.16:4294 , piece_id=HLYZFPCCGUP3RSECY4NMMY6JDO6FQS3RS33CKGSLFOCXLSDKP6UA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:10:18.490272690 (1h18m34.271206075s) [noise=true , address=104.36.82.14:4223 , piece_id=RPJB2O24KCERXUNZ4DJQEI3JFEAWQDLIN3U3IVW57QIYGP5MQFWA, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:10:18.503742003 (1h18m34.284675381s) [noise=true , address=104.36.82.6:4874 , piece_id=PZY35ASQBO7MRVGJF6OMON3AHHEXRXSPRUTBFXZFI5LIBVLJNSUQ, seg_pos={0 0}, upload=1204]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:10:18.503784922 (1h18m34.284718311s) [seg_pos={0 0}, upload=1204]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:10:18.531652200 (1h18m34.312585589s) [upload=1204]: Done uploading segments... Jul 26 23:10:18.531687243 (1h18m34.312620624s) [upload=1204]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.046156275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X................X........................X......XXXX] duration: 30.088130776s FAIL (2025-07-26 23:10:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.08818312s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 418.968648ms FAIL (2025-07-26 23:10:02 UTC) orbiter warning: uplink: too many requests Duration: 419.003122ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...................................................XXX] duration: 30.030610668s FAIL (2025-07-26 23:09:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ]: Got next segment. Inline: false Jul 26 23:09:18.432371674 (1h17m34.213305054s) [upload=1193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:09:18.432377637 (1h17m34.213311016s) [upload=1193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:09:18.465811046 (1h17m34.246744434s) [seg_pos={0 0}, upload=1193]: Begin upload segment... Jul 26 23:09:18.465867150 (1h17m34.246800534s) [seg_pos={0 0}, upload=1193]: Done begin upload segment. Jul 26 23:09:18.465896758 (1h17m34.246830142s) [noise=true , address=45.91.116.4:4937 , piece_id=PGOJONJ5CHQCGA5C54XV7JSJU6RM3ILANHRR4YYRQIX7X2NEPDPA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.466458760 (1h17m34.247392143s) [noise=true , address=104.36.82.4:4804 , piece_id=W7QOVFNTAFEVPPD6JJ6X6SYOPGKCAW4V65Q2EZHBTY34REYETYNA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.466863253 (1h17m34.247796633s) [noise=true , address=104.36.82.11:4115 , piece_id=H6EKTXQVW65GJSUO4IHRXSBPBYTNYULRTZVC4PA4LRTQC7UIDYEA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.467293032 (1h17m34.248226417s) [noise=true , address=77.90.31.17:4770 , piece_id=PAER4IRAVI7JOBRRMRGEYI6NZKBAKMB3TTLEBYPGURY2KWU2DYMQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.467669063 (1h17m34.248602443s) [noise=true , address=104.36.82.8:4969 , piece_id=6OMJWEDFRNVYFUMN74RSYBCJJBAVKZJWNB2GEXCWHU62B6SSW4EQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.468059548 (1h17m34.248992929s) [noise=true , address=45.91.116.14:4191 , piece_id=7AX6IAPXVGI6LRWPWYFDOB3XRG3IU55MLNNK5JNHESA6WBJ6CPBQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.468590945 (1h17m34.249524329s) [noise=true , address=45.91.116.14:4190 , piece_id=YX5GGZMX7DLCQF4UBLHWVAINKAYPZ5H3RGH67LJQRL4IFSRSYWZQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.469323984 (1h17m34.250257365s) [noise=true , address=45.91.116.10:4082 , piece_id=YWLO2R2GNZBUKQ4ZXYTT43YIRKHGPKTYCNI7CERULUUKPISV5XHA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.469876574 (1h17m34.25080996s) [noise=true , address=104.36.82.14:4223 , piece_id=HQVUTLZWHJQINARA6JONTFZOUT3YRR2X6JVGLUSK3FLHYWFX7PIQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.470344883 (1h17m34.251278269s) [noise=true , address=45.91.116.6:4975 , piece_id=R4LFIY35FO3MH6WBVHKDJAF54QTRRBGS775JRS6JKMRJVAPZJLDA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.470755100 (1h17m34.251688485s) [noise=true , address=45.91.116.8:4047 , piece_id=XZRBQ4FV6LW2W76LIN5ZDDWOPQEPSM6QIQHVMGVLHMTFMFACASAA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.471146128 (1h17m34.252079518s) [noise=true , address=104.36.82.15:4257 , piece_id=7DJOAROPEFSJBKGYNQFMEKPROUCDYEHXHK7DSP4BV5V76KWVORAQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.471796301 (1h17m34.25272969s) [noise=true , address=104.36.82.16:4295 , piece_id=ZDOVLWZHHQQOGUAVDZMPRTNRCUN6HVGKMOPFAZ4KAZJMD4YYJCFQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.471872781 (1h17m34.252806166s) [noise=true , address=104.36.82.2:4729 , piece_id=GE6X37UTZNRSS5KV3L6OUYFEMSVNIFZTBFWCB5QPENIQWBADM7PQ, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.472443650 (1h17m34.253377033s) [noise=true , address=45.91.116.5:4298 , piece_id=N6IJDS3YDAWNKHJ2JJDAIP2PSPGHJG24S2JR7BIN3ZO26ZC6QLJA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.472917278 (1h17m34.25385066s) [noise=true , address=104.36.82.2:4730 , piece_id=GOJFECRKJGNI4QQEUPRL5GJEAZIEOYGR3KPRUH56V5SPWRLBLGXA, seg_pos={0 0}, upload=1193]: Uploading piece... Jul 26 23:09:18.597570022 (1h17m34.378503385s) [upload=1193]: Next returned nil segment Jul 26 23:09:18.597589799 (1h17m34.378523178s) [upload=1193]: Waiting for error group managing segments... Jul 26 23:09:18.601719002 (1h17m34.382652389s) [noise=true , address=45.91.116.5:4298 , piece_id=N6IJDS3YDAWNKHJ2JJDAIP2PSPGHJG24S2JR7BIN3ZO26ZC6QLJA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.602366275 (1h17m34.383299639s) [noise=true , address=45.91.116.14:4191 , piece_id=7AX6IAPXVGI6LRWPWYFDOB3XRG3IU55MLNNK5JNHESA6WBJ6CPBQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.602796132 (1h17m34.383729497s) [noise=true , address=45.91.116.8:4047 , piece_id=XZRBQ4FV6LW2W76LIN5ZDDWOPQEPSM6QIQHVMGVLHMTFMFACASAA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.602931699 (1h17m34.383865071s) [noise=true , address=45.91.116.4:4937 , piece_id=PGOJONJ5CHQCGA5C54XV7JSJU6RM3ILANHRR4YYRQIX7X2NEPDPA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.604654573 (1h17m34.385587943s) [noise=true , address=45.91.116.10:4082 , piece_id=YWLO2R2GNZBUKQ4ZXYTT43YIRKHGPKTYCNI7CERULUUKPISV5XHA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.607338568 (1h17m34.388272023s) [noise=true , address=45.91.116.6:4975 , piece_id=R4LFIY35FO3MH6WBVHKDJAF54QTRRBGS775JRS6JKMRJVAPZJLDA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:18.608292569 (1h17m34.389225951s) [noise=true , address=45.91.116.14:4190 , piece_id=YX5GGZMX7DLCQF4UBLHWVAINKAYPZ5H3RGH67LJQRL4IFSRSYWZQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Jul 26 23:09:48.394187362 (1h18m4.175120847s) [noise=true , address=104.36.82.2:4730 , piece_id=GOJFECRKJGNI4QQEUPRL5GJEAZIEOYGR3KPRUH56V5SPWRLBLGXA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:35092->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394268959 (1h18m4.175202341s) [noise=true , address=104.36.82.8:4969 , piece_id=6OMJWEDFRNVYFUMN74RSYBCJJBAVKZJWNB2GEXCWHU62B6SSW4EQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:42764->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394289525 (1h18m4.175222901s) [noise=true , address=77.90.31.17:4770 , piece_id=PAER4IRAVI7JOBRRMRGEYI6NZKBAKMB3TTLEBYPGURY2KWU2DYMQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 159.223.143.39:58162->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394358473 (1h18m4.175291857s) [noise=true , address=104.36.82.15:4257 , piece_id=7DJOAROPEFSJBKGYNQFMEKPROUCDYEHXHK7DSP4BV5V76KWVORAQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:52150->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394420352 (1h18m4.175353732s) [noise=true , address=104.36.82.11:4115 , piece_id=H6EKTXQVW65GJSUO4IHRXSBPBYTNYULRTZVC4PA4LRTQC7UIDYEA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:49834->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394432927 (1h18m4.175366296s) [noise=true , address=104.36.82.2:4729 , piece_id=GE6X37UTZNRSS5KV3L6OUYFEMSVNIFZTBFWCB5QPENIQWBADM7PQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:56974->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394455425 (1h18m4.175388814s) [noise=true , address=104.36.82.16:4295 , piece_id=ZDOVLWZHHQQOGUAVDZMPRTNRCUN6HVGKMOPFAZ4KAZJMD4YYJCFQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:43120->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394469618 (1h18m4.17540299s) [noise=true , address=104.36.82.4:4804 , piece_id=W7QOVFNTAFEVPPD6JJ6X6SYOPGKCAW4V65Q2EZHBTY34REYETYNA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:52334->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:48.394579056 (1h18m4.175512433s) [noise=true , address=104.36.82.14:4223 , piece_id=HQVUTLZWHJQINARA6JONTFZOUT3YRR2X6JVGLUSK3FLHYWFX7PIQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:09:48.394614130 (1h18m4.175547518s) [seg_pos={0 0}, upload=1193]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:09:48.421850894 (1h18m4.20278427s) [upload=1193]: Done uploading segments... Jul 26 23:09:48.421882670 (1h18m4.202816041s) [upload=1193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.030644997s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 553.73248ms FAIL (2025-07-26 23:09:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.766025ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXX] duration: 1m0.040049058s FAIL (2025-07-26 23:09:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 57.422906385s) [noise=true , address=104.36.82.8:4983 , piece_id=R72O7V53TGWQJWOFIULAHEMKNTKDHH4M3T5RTDM67RMWPQVXPGUQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.642277141 (1h16m57.423210521s) [noise=true , address=104.36.82.52:4111 , piece_id=GSSC7SCP2V7GA4V3AI2CSPW6W5G2FE3KTCZOMUMOK4BDS2VRDREQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.642580559 (1h16m57.423513944s) [noise=true , address=104.36.82.55:4196 , piece_id=NFMMO4HJJEBAQOSOZKXMI4MB2AZYIQWXVCJMJJ2MVVR4TNGZYOIQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.642858874 (1h16m57.42379225s) [noise=true , address=77.90.31.21:4891 , piece_id=IEBN3BZ5SX4XSU5NETOOBYY3RISZULT6236IDSO2BMHCELMG75LQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.645910284 (1h16m57.426843671s) [noise=true , address=79.127.220.101:4048 , piece_id=HSONGOMNWANJOGGLR53W2CX3I3UW77FVHIUMVFUGAMPORUCTSGZA, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.646008935 (1h16m57.426942322s) [noise=true , address=79.127.205.234:4616 , piece_id=HZZDM6BY74275S6OGNYHYR2A7UWT66PXPQXS3VP74HXSUSNNHAEA, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.646496406 (1h16m57.427429836s) [noise=true , address=104.36.82.49:4991 , piece_id=6V4QTR7T7ZA74GZX4XW7JRIRXH4HZM733ALNOIXR4NZI6A3FNLZA, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.646920477 (1h16m57.427853865s) [noise=true , address=104.36.82.47:4530 , piece_id=HE6UUO3GPVXL43ZE6IVJWISUU65IG7NLAX7ODO4BJ5HQAIIAFZDQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.647365862 (1h16m57.428299251s) [noise=true , address=104.36.82.54:4179 , piece_id=RHIS5OGTBELYNDGO37SVEKAIXQ4KQ4PXWO4BVCE7DOX5CMBWBKHA, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.647785524 (1h16m57.428718904s) [noise=true , address=79.127.219.38:4015 , piece_id=VHXLI2VBRPUGIMAPDYFTEMO7AIDUNH3HVRTD4W343DYT6G5WJG7Q, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.648448970 (1h16m57.429382359s) [noise=true , address=104.36.82.56:4249 , piece_id=OEDELWTGFVCU6LDMLHMFXCOXE6WG34R4YPPJKOXYOP2QYW6F3FAA, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.648739076 (1h16m57.429672466s) [noise=true , address=104.36.82.4:4808 , piece_id=7FOTM4I2QZR7FKNHLGSIXVBHRAMVLZCOP2VZBNDPKFW5IQ7GOUUQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.649141483 (1h16m57.43007487s) [noise=true , address=104.36.82.46:4496 , piece_id=RMHX4PMWRSF7X5CBVFF2F6DEP7AFSZGTIDRY6QVFEFCS63G4IP5Q, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.649602078 (1h16m57.430535465s) [noise=true , address=104.36.82.22:4525 , piece_id=VIOBZYCIDOPVGI7JQTXO4UBFRKTWM2C6FWZ4OXBINEJR2ELGBXUQ, seg_pos={0 0}, upload=1187]: Uploading piece... Jul 26 23:08:41.742258255 (1h16m57.523191639s) [upload=1187]: Next returned nil segment Jul 26 23:08:41.742278921 (1h16m57.523212305s) [upload=1187]: Waiting for error group managing segments... Jul 26 23:08:41.746632139 (1h16m57.527565508s) [noise=true , address=79.127.219.38:4015 , piece_id=VHXLI2VBRPUGIMAPDYFTEMO7AIDUNH3HVRTD4W343DYT6G5WJG7Q, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:08:41.748561266 (1h16m57.529494644s) [noise=true , address=79.127.219.47:4048 , piece_id=VBDO3WAUUDMDSBSMGJG56JCBWPWCRNOHGZPPCLMJ64DGGKZGDGGQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:08:41.770647086 (1h16m57.551580549s) [noise=true , address=77.90.31.21:4891 , piece_id=IEBN3BZ5SX4XSU5NETOOBYY3RISZULT6236IDSO2BMHCELMG75LQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:08:41.837884657 (1h16m57.618818027s) [noise=true , address=79.127.220.101:4048 , piece_id=HSONGOMNWANJOGGLR53W2CX3I3UW77FVHIUMVFUGAMPORUCTSGZA, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:08:41.977574807 (1h16m57.758508185s) [noise=true , address=79.127.201.215:4343 , piece_id=PFY3KNLG3MYKKQ5NRXSMTYNWCLWENKJRXTBHBRQGLPE2P4BNSP4A, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:08:42.251207419 (1h16m58.03214079s) [noise=true , address=79.127.205.234:4616 , piece_id=HZZDM6BY74275S6OGNYHYR2A7UWT66PXPQXS3VP74HXSUSNNHAEA, seg_pos={0 0}, upload=1187]: Done uploading piece. err: <nil> Jul 26 23:09:41.573331311 (1h17m57.354264701s) [noise=true , address=104.36.82.52:4111 , piece_id=GSSC7SCP2V7GA4V3AI2CSPW6W5G2FE3KTCZOMUMOK4BDS2VRDREQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.573417694 (1h17m57.354351071s) [noise=true , address=104.36.82.54:4179 , piece_id=RHIS5OGTBELYNDGO37SVEKAIXQ4KQ4PXWO4BVCE7DOX5CMBWBKHA, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: rpc: dial tcp 104.36.82.54:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.573669940 (1h17m57.354603326s) [noise=true , address=104.36.82.56:4249 , piece_id=OEDELWTGFVCU6LDMLHMFXCOXE6WG34R4YPPJKOXYOP2QYW6F3FAA, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: rpc: dial tcp 104.36.82.56:4249: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.573745315 (1h17m57.354678693s) [noise=true , address=104.36.82.53:4135 , piece_id=ZOCEYXE5WRVWON6VOH7OBD4F4FGOHTOWKZEX6W4UFZGPHYOIMULQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.573805928 (1h17m57.35473931s) [noise=true , address=104.36.82.55:4196 , piece_id=NFMMO4HJJEBAQOSOZKXMI4MB2AZYIQWXVCJMJJ2MVVR4TNGZYOIQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.574616993 (1h17m57.355550356s) [noise=true , address=104.36.82.6:4888 , piece_id=6OIKQVGIGJVXD7SLKH5CEJQ276MXYNICUTOMKRJUOSOI4GMGAIYQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12HwYszCTcXEHqSNnWJpT11NpKg8cFUvdAqU2wUigsLbxmDFkg6, address:104.36.82.6:4888): protocol: rpc: dial tcp 104.36.82.6:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.574706583 (1h17m57.355639964s) [noise=true , address=104.36.82.47:4530 , piece_id=HE6UUO3GPVXL43ZE6IVJWISUU65IG7NLAX7ODO4BJ5HQAIIAFZDQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.574772328 (1h17m57.355705706s) [noise=true , address=104.36.82.22:4525 , piece_id=VIOBZYCIDOPVGI7JQTXO4UBFRKTWM2C6FWZ4OXBINEJR2ELGBXUQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12Yx3qg5NAW8iAw56mkwALjkkhyFF5feKgTZxcTG31RxBsubh5A, address:104.36.82.22:4525): protocol: write tcp 159.223.143.39:42568->104.36.82.22:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.574833085 (1h17m57.355766452s) [noise=true , address=104.36.82.49:4991 , piece_id=6V4QTR7T7ZA74GZX4XW7JRIRXH4HZM733ALNOIXR4NZI6A3FNLZA, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:1PQti6scwGXWwevzaBZMDqgR6WFz3NRNdBjw8eF59NZq15VGjA, address:104.36.82.49:4991): protocol: rpc: dial tcp 104.36.82.49:4991: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.574888243 (1h17m57.355821606s) [noise=true , address=104.36.82.8:4983 , piece_id=R72O7V53TGWQJWOFIULAHEMKNTKDHH4M3T5RTDM67RMWPQVXPGUQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:104.36.82.8:4983): protocol: rpc: dial tcp 104.36.82.8:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.575144756 (1h17m57.356078137s) [noise=true , address=104.36.82.46:4496 , piece_id=RMHX4PMWRSF7X5CBVFF2F6DEP7AFSZGTIDRY6QVFEFCS63G4IP5Q, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: rpc: dial tcp 104.36.82.46:4496: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.575215786 (1h17m57.356149149s) [noise=true , address=104.36.82.4:4808 , piece_id=7FOTM4I2QZR7FKNHLGSIXVBHRAMVLZCOP2VZBNDPKFW5IQ7GOUUQ, seg_pos={0 0}, upload=1187]: Done uploading piece. err: ecclient: upload failed (node:1fbQCaGY9ZssBzyNusnhyY994e8ytvW4sQDQRt7q2tdGrg6ps3, address:104.36.82.4:4808): protocol: rpc: dial tcp 104.36.82.4:4808: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:41.575250726 (1h17m57.356184113s) [seg_pos={0 0}, upload=1187]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:09:41.610657295 (1h17m57.391590665s) [upload=1187]: Done uploading segments... Jul 26 23:09:41.610687598 (1h17m57.391620979s) [upload=1187]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.040076956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 452.843471ms FAIL (2025-07-26 23:09:28 UTC) orbiter warning: uplink: too many requests Duration: 452.879852ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X................X........................X......XXX] duration: 30.044286121s FAIL (2025-07-26 23:09:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.044375747s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.002821063s FAIL (2025-07-26 23:09:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002844628s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...X.X.X...XX.................X.X..........XX...X...X..XX.X.] duration: 12.131938687s FAIL (2025-07-26 23:09:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.131960977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 219.685501ms FAIL (2025-07-26 23:08:56 UTC) orbiter warning: uplink: too many requests Duration: 219.740822ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....X.X.X...XX.................X.X..........XX...X...X..XX.X] duration: 1m0.001396893s FAIL (2025-07-26 23:08:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001576066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................................................XX] duration: 30.046549474s FAIL (2025-07-26 23:08:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 97685931 (1h16m33.578619318s) [upload=1181]: Starting upload Jul 26 23:08:17.797842115 (1h16m33.578775501s) [upload=1181]: Done starting upload Jul 26 23:08:17.798039081 (1h16m33.578972452s) [upload=1181]: Uploading segments... Jul 26 23:08:17.798473555 (1h16m33.57940692s) [upload=1181]: Got next segment. Inline: false Jul 26 23:08:17.798565517 (1h16m33.579498897s) [upload=1181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:08:17.798572280 (1h16m33.579505656s) [upload=1181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:08:17.824915523 (1h16m33.6058489s) [seg_pos={0 0}, upload=1181]: Begin upload segment... Jul 26 23:08:17.824969912 (1h16m33.605903288s) [seg_pos={0 0}, upload=1181]: Done begin upload segment. Jul 26 23:08:17.824997041 (1h16m33.60593042s) [noise=true , address=45.91.116.7:4010 , piece_id=N66J3MYD4QBVDMDOP3NNTQVGMSJCTWXHY34NS3H7JRE6T3SBDIWQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.825563272 (1h16m33.606496661s) [noise=true , address=104.36.82.8:4969 , piece_id=L5F6DW4B2W5EQW7THR4O53SMBQAPNTWW45BD6KV6VB45IWMA6ZDA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.826673992 (1h16m33.607607379s) [noise=true , address=45.91.116.4:4938 , piece_id=TV2GG5UHK3E4HIUKVBCCZGKPBKWBCCS6JOZBLOG3EIXQG6GMM3NA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827185284 (1h16m33.608118711s) [noise=true , address=104.36.82.14:4223 , piece_id=P5L2RLUSGY375RHA7HNLKEVOF5QLCRIHY6F57DJI5MSQFVQ7VGXQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827636178 (1h16m33.608569545s) [noise=true , address=104.36.82.15:4259 , piece_id=OPTCVANC4E5OWF35VIS7NMTWNVYC4B6MYYSAIIBTUFSAPXWBAUWA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827966744 (1h16m33.608900126s) [noise=true , address=104.36.82.4:4804 , piece_id=ET4V42VZ3XOX6X2LS4F2VMQOKU5ATGNMESSI2WLHFWQQ42A4OBJA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.828361630 (1h16m33.609295007s) [noise=true , address=104.36.82.9:4007 , piece_id=IN6MWO2P2A7M64KSBXZALRKCSP46OLFDUKV73WCQGMGZFSXG6QVA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.828907321 (1h16m33.609840707s) [noise=true , address=104.36.82.2:4732 , piece_id=EL5YIXTMUJEKXGFTJ5ZHTQZUJW4ELYSBQ4YDDCMSTSOTCXVM37MQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829116385 (1h16m33.610049765s) [noise=true , address=45.91.116.11:4118 , piece_id=LFGDECRUIOJLRPUO5NWOVQNMG6SKQVUT4RY6QPBPGKOVHROBVPQQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829419025 (1h16m33.610352405s) [noise=true , address=104.36.82.3:4765 , piece_id=OIFXWMKVREKJVSRKY6KQRVKIEQRIOSQBT7AR72T67LNUECD4ALCA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829504495 (1h16m33.610437883s) [noise=true , address=77.90.31.14:4663 , piece_id=R4NQENA2VJEDN3HV6X4LYWELNHWEI5TGJFU3GVCCRZ5LKEMUZ3BQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829923294 (1h16m33.610856675s) [noise=true , address=45.91.116.8:4048 , piece_id=SPTOI42AXO5S666EWMRAQMEPQ4CLLMDV7SP2FW5WISCFLIHCQHNQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.830391997 (1h16m33.611325385s) [noise=true , address=45.91.116.4:4940 , piece_id=LWV5TAY5XETIWZRO3MQZEVH7HZUJKQ52QJ5T43MGNYARSHCUOLHQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.830655447 (1h16m33.611588826s) [noise=true , address=104.36.82.8:4972 , piece_id=32QBA4R5S4APFXWMJLKEZCDVMOIJ4LXOZATGGQUVBKAXEGWVFY6A, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.831648357 (1h16m33.61258175s) [noise=true , address=104.36.82.6:4876 , piece_id=6FIJW7ZPY2BY2L374UUXQ563SMSWFJMNB4VB5X5WNZ55MAAU7WZA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.832405120 (1h16m33.613338533s) [noise=true , address=45.91.116.12:4227 , piece_id=PEBKEZBE2KV5IGCTGI222OCYI6NO7UPVJX66V3JZVOXL32XCVNMQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.915119182 (1h16m33.696052559s) [upload=1181]: Next returned nil segment Jul 26 23:08:17.915141507 (1h16m33.696074892s) [upload=1181]: Waiting for error group managing segments... Jul 26 23:08:17.919860312 (1h16m33.700793696s) [noise=true , address=45.91.116.7:4010 , piece_id=N66J3MYD4QBVDMDOP3NNTQVGMSJCTWXHY34NS3H7JRE6T3SBDIWQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.921371973 (1h16m33.702305354s) [noise=true , address=45.91.116.11:4118 , piece_id=LFGDECRUIOJLRPUO5NWOVQNMG6SKQVUT4RY6QPBPGKOVHROBVPQQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.921938398 (1h16m33.702871761s) [noise=true , address=45.91.116.8:4048 , piece_id=SPTOI42AXO5S666EWMRAQMEPQ4CLLMDV7SP2FW5WISCFLIHCQHNQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.922448973 (1h16m33.703382335s) [noise=true , address=45.91.116.4:4940 , piece_id=LWV5TAY5XETIWZRO3MQZEVH7HZUJKQ52QJ5T43MGNYARSHCUOLHQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.923088658 (1h16m33.70402202s) [noise=true , address=45.91.116.4:4938 , piece_id=TV2GG5UHK3E4HIUKVBCCZGKPBKWBCCS6JOZBLOG3EIXQG6GMM3NA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.923644261 (1h16m33.704577647s) [noise=true , address=45.91.116.12:4227 , piece_id=PEBKEZBE2KV5IGCTGI222OCYI6NO7UPVJX66V3JZVOXL32XCVNMQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.929203579 (1h16m33.710136962s) [noise=true , address=77.90.31.14:4663 , piece_id=R4NQENA2VJEDN3HV6X4LYWELNHWEI5TGJFU3GVCCRZ5LKEMUZ3BQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:47.760958504 (1h17m3.541891893s) [noise=true , address=104.36.82.14:4223 , piece_id=P5L2RLUSGY375RHA7HNLKEVOF5QLCRIHY6F57DJI5MSQFVQ7VGXQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761069173 (1h17m3.542002537s) [noise=true , address=104.36.82.15:4259 , piece_id=OPTCVANC4E5OWF35VIS7NMTWNVYC4B6MYYSAIIBTUFSAPXWBAUWA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761152042 (1h17m3.542085416s) [noise=true , address=104.36.82.3:4765 , piece_id=OIFXWMKVREKJVSRKY6KQRVKIEQRIOSQBT7AR72T67LNUECD4ALCA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761324310 (1h17m3.542257691s) [noise=true , address=104.36.82.6:4876 , piece_id=6FIJW7ZPY2BY2L374UUXQ563SMSWFJMNB4VB5X5WNZ55MAAU7WZA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:08:47.763634370 (1h17m3.544567761s) [noise=true , address=104.36.82.8:4969 , piece_id=L5F6DW4B2W5EQW7THR4O53SMBQAPNTWW45BD6KV6VB45IWMA6ZDA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.763764969 (1h17m3.544698334s) [noise=true , address=104.36.82.9:4007 , piece_id=IN6MWO2P2A7M64KSBXZALRKCSP46OLFDUKV73WCQGMGZFSXG6QVA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.763856186 (1h17m3.544789564s) [noise=true , address=104.36.82.8:4972 , piece_id=32QBA4R5S4APFXWMJLKEZCDVMOIJ4LXOZATGGQUVBKAXEGWVFY6A, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:46250->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.764028239 (1h17m3.544961611s) [noise=true , address=104.36.82.4:4804 , piece_id=ET4V42VZ3XOX6X2LS4F2VMQOKU5ATGNMESSI2WLHFWQQ42A4OBJA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.764270783 (1h17m3.545204146s) [noise=true , address=104.36.82.2:4732 , piece_id=EL5YIXTMUJEKXGFTJ5ZHTQZUJW4ELYSBQ4YDDCMSTSOTCXVM37MQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:51900->104.36.82.2:4732: use of closed network connection; conn closed: op Jul 26 23:08:47.764301402 (1h17m3.545234793s) [seg_pos={0 0}, upload=1181]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:47.798682499 (1h17m3.579615873s) [upload=1181]: Done uploading segments... Jul 26 23:08:47.798712233 (1h17m3.579645833s) [upload=1181]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.046575849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXX] duration: 1m0.038879132s FAIL (2025-07-26 23:08:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 1]: Uploading piece... Jul 26 23:07:40.682030269 (1h15m56.462963648s) [noise=true , address=104.36.82.47:4541 , piece_id=HF3L73Q6NTSUH4HMAGJAVHW4T3UNHB6XBBZWFAW6XJQIY4HKQZBA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682294597 (1h15m56.463227981s) [noise=true , address=104.36.82.42:4805 , piece_id=2JBJZS5KD2ULEKX2JAHTK466WH5EHH6KDLJJPCGTWX2NTPH3VQSQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682577978 (1h15m56.463511371s) [noise=true , address=104.36.82.3:4784 , piece_id=6AMRXJOLZUUCTD3KSO3COBIMLSCJGMTY3ER573VLFNP6D42C5NUA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682859361 (1h15m56.463792734s) [noise=true , address=104.36.82.46:4508 , piece_id=2QXA2OCHDHMIYI2FXHWOJLLOLHA2TEHWWTZEZV45QADEN23NXDAQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.683313077 (1h15m56.464246464s) [noise=true , address=79.127.205.233:4650 , piece_id=DG2KEGNOEFZUKIGKZ5VPGXMC7YSCSYUSTSXY4DTBV6Q6WEOLKZRQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.709755308 (1h15m56.490688696s) [noise=true , address=104.36.82.40:4953 , piece_id=UIUPAX6A7ABU3ZAFQVIIIYGQJCRZE5JOHTDUF4CYOEABN4EMTJQA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.711181562 (1h15m56.49211496s) [noise=true , address=104.36.82.53:4133 , piece_id=4FU4UFMDXNTUQZ2TIMSEA6VPLGPXTFQBO3A7RUYSU2LTXBWHM45A, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.711652969 (1h15m56.492586357s) [noise=true , address=104.36.82.36:4650 , piece_id=4RONA555PXK2FQS5ERFID5MXAEYSS65JGFHK5QRWDLVXDRWPLKDA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712089569 (1h15m56.493022956s) [noise=true , address=104.36.82.35:4612 , piece_id=DD6YHE77ZZWYSWKX2MAU7OUM6OJP7JPJWYVMNRWZO7O4PSECFHUA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712510112 (1h15m56.493443504s) [noise=true , address=104.36.82.44:4885 , piece_id=CX7I6JHDTUNXPUSST7UCC6X6PCWZGEDMWIUVJRJLYR2FRUCHGP7A, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712906845 (1h15m56.493840213s) [noise=true , address=104.36.82.21:4485 , piece_id=FDWUOES5NG5LIZCNKRDW5DRSXMZ4EM53WBW3564N5WBCJPHQ5XFA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.713257134 (1h15m56.494190522s) [noise=true , address=77.90.31.13:4658 , piece_id=F4SXB3B2TV57K2WTI6YVOSYQRKPU2DOZ3PU6SSB2JCB2TJSW3YBQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.713609748 (1h15m56.49454313s) [noise=true , address=104.36.82.56:4264 , piece_id=JWOHBFMD3YG2LFHMQQZQ3E7UQ43HPDERKH6MIJNNNM4O3DP43CBA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.815609578 (1h15m56.596542947s) [upload=1171]: Next returned nil segment Jul 26 23:07:40.815625479 (1h15m56.596558848s) [upload=1171]: Waiting for error group managing segments... Jul 26 23:07:40.822002686 (1h15m56.602936055s) [noise=true , address=79.127.219.38:4018 , piece_id=7DKMGNGVS27SKALWMZJ3XE7LMPIRLAZ3MXR2KYSQM6VA2AIA36NQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.834523370 (1h15m56.615456751s) [noise=true , address=104.36.82.52:4101 , piece_id=GVNIIJGYFBY2FIJVZX7L5MH4YV3QMVZ4LMTUWHQSDOT6PIEOTLOQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.836884698 (1h15m56.617818081s) [noise=true , address=104.36.82.56:4264 , piece_id=JWOHBFMD3YG2LFHMQQZQ3E7UQ43HPDERKH6MIJNNNM4O3DP43CBA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.841110406 (1h15m56.622043786s) [noise=true , address=104.36.82.53:4133 , piece_id=4FU4UFMDXNTUQZ2TIMSEA6VPLGPXTFQBO3A7RUYSU2LTXBWHM45A, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:41.323408089 (1h15m57.104341472s) [noise=true , address=79.127.205.233:4650 , piece_id=DG2KEGNOEFZUKIGKZ5VPGXMC7YSCSYUSTSXY4DTBV6Q6WEOLKZRQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:47.827730169 (1h16m3.608663546s) [noise=true , address=104.36.82.54:4168 , piece_id=CJBMJASUWS74DKCYLZEYBGWTZ6CBJYY4YOPHNQ2YF4R3GKSMISXA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:08:40.610534953 (1h16m56.391468339s) [noise=true , address=104.36.82.42:4805 , piece_id=2JBJZS5KD2ULEKX2JAHTK466WH5EHH6KDLJJPCGTWX2NTPH3VQSQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: rpc: dial tcp 104.36.82.42:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610647891 (1h16m56.391581276s) [noise=true , address=104.36.82.3:4784 , piece_id=6AMRXJOLZUUCTD3KSO3COBIMLSCJGMTY3ER573VLFNP6D42C5NUA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3, address:104.36.82.3:4784): protocol: rpc: dial tcp 104.36.82.3:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610733661 (1h16m56.391667041s) [noise=true , address=104.36.82.41:4785 , piece_id=YP77FRE3L2SRTNKP43PMEXIUR2EG2WJSL3QUA2Z45CXHCFKW66QQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB, address:104.36.82.41:4785): protocol: rpc: dial tcp 104.36.82.41:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610796410 (1h16m56.391729789s) [noise=true , address=104.36.82.40:4953 , piece_id=UIUPAX6A7ABU3ZAFQVIIIYGQJCRZE5JOHTDUF4CYOEABN4EMTJQA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: rpc: dial tcp 104.36.82.40:4953: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610821283 (1h16m56.391754662s) [noise=true , address=104.36.82.47:4541 , piece_id=HF3L73Q6NTSUH4HMAGJAVHW4T3UNHB6XBBZWFAW6XJQIY4HKQZBA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610887841 (1h16m56.39182122s) [noise=true , address=104.36.82.44:4885 , piece_id=CX7I6JHDTUNXPUSST7UCC6X6PCWZGEDMWIUVJRJLYR2FRUCHGP7A, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: rpc: dial tcp 104.36.82.44:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610941206 (1h16m56.391874588s) [noise=true , address=104.36.82.21:4485 , piece_id=FDWUOES5NG5LIZCNKRDW5DRSXMZ4EM53WBW3564N5WBCJPHQ5XFA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1wmohWyFaMkNm8KBAZprmZ6RuiH6UcB8KRNght2HGpV3Ucnnq9, address:104.36.82.21:4485): protocol: write tcp 159.223.143.39:58518->104.36.82.21:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610951572 (1h16m56.39188494s) [noise=true , address=104.36.82.36:4650 , piece_id=4RONA555PXK2FQS5ERFID5MXAEYSS65JGFHK5QRWDLVXDRWPLKDA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1FM1g8LyQ9LJCNjKLxJtTQyDNudVcN4tv3NsxKRTV5gVChjCzp, address:104.36.82.36:4650): protocol: write tcp 159.223.143.39:43804->104.36.82.36:4650: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610965687 (1h16m56.391899066s) [noise=true , address=104.36.82.22:4538 , piece_id=3TH2UZP6QEODNVFFDSAXZZ7KQEBC2ZSELBXPCOKZKXU45VKDD3YQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:104.36.82.22:4538): protocol: write tcp 159.223.143.39:44462->104.36.82.22:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611083592 (1h16m56.392016972s) [noise=true , address=77.90.31.13:4658 , piece_id=F4SXB3B2TV57K2WTI6YVOSYQRKPU2DOZ3PU6SSB2JCB2TJSW3YBQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:57360->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611097271 (1h16m56.392030641s) [noise=true , address=104.36.82.35:4612 , piece_id=DD6YHE77ZZWYSWKX2MAU7OUM6OJP7JPJWYVMNRWZO7O4PSECFHUA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: rpc: dial tcp 104.36.82.35:4612: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611123970 (1h16m56.392057349s) [noise=true , address=104.36.82.46:4508 , piece_id=2QXA2OCHDHMIYI2FXHWOJLLOLHA2TEHWWTZEZV45QADEN23NXDAQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: read tcp 159.223.143.39:38136->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611156836 (1h16m56.3920902s) [seg_pos={0 0}, upload=1171]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:08:40.644393326 (1h16m56.425326708s) [upload=1171]: Done uploading segments... Jul 26 23:08:40.644441483 (1h16m56.425374862s) [upload=1171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.038903503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.002447769s FAIL (2025-07-26 23:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002467159s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 30.032342542s FAIL (2025-07-26 23:08:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 23:07:46.502507465 (1h16m2.283440843s) [upload=1175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:07:46.502514698 (1h16m2.283448083s) [upload=1175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:07:46.538591482 (1h16m2.319524896s) [seg_pos={0 0}, upload=1175]: Begin upload segment... Jul 26 23:07:46.538666369 (1h16m2.319599753s) [seg_pos={0 0}, upload=1175]: Done begin upload segment. Jul 26 23:07:46.538988284 (1h16m2.319921668s) [noise=true , address=104.36.82.6:4875 , piece_id=MUBYCAFZZNFAWUZDOZZQIKAFQSIZWF7AEEZRF47INQ2EAF66NNZA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.539079551 (1h16m2.320012938s) [noise=true , address=104.36.82.15:4257 , piece_id=N7UGWSCY4LR5TPC5RED7YDIW5TIZMXPZ35SUWTVPIZTRR6BFMDMQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.539804658 (1h16m2.320738047s) [noise=true , address=104.36.82.2:4729 , piece_id=XXZNOYSYRIIJIR54NTZXL7F7XXYIPCLJY4X6NGV2VLSD43UD2QFQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541071101 (1h16m2.322004499s) [noise=true , address=104.36.82.14:4223 , piece_id=YQFOFJBGZMDVNGIAU7NR5HWMNN4SXWKKLNRFRYMMW2O6QUGA7SKQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541194964 (1h16m2.322128347s) [noise=true , address=104.36.82.8:4970 , piece_id=WIZILE4HKCT4EBR7NO6USMQLVH3ANCRKXUIMDSRGK2W6BZNAOJGQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541735822 (1h16m2.322669208s) [noise=true , address=45.91.116.5:4297 , piece_id=QOL343MXQK3D5PXHWAAXVJJXUVVRDSRXOYUGDXUWA6E6NFW5BW6A, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.542375818 (1h16m2.323309208s) [noise=true , address=45.91.116.4:4938 , piece_id=MVZ2K4WJEN4GNWCO2RXE6XCF7Y7YXWV5UVD6K4PPEIKUMK5JG7PQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.542768507 (1h16m2.323701895s) [noise=true , address=45.91.116.8:4047 , piece_id=AVYWRB5JKMKRVUBM7CONKSFLWEVBMKOCHRV3ERSXNVTPB6VNFAEQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.543380518 (1h16m2.324313908s) [noise=true , address=104.36.82.15:4258 , piece_id=MSSR4RXTG4RQMJR6DWE2IISGHEHURQGIZUJ3TNZW5XX64ZGM3RLA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.544602822 (1h16m2.325536213s) [noise=true , address=104.36.82.4:4802 , piece_id=WOP737BTCQ3G6NUWIGKRRZR6KYMPHN7WUJ5HOPZTNYHK45K77H7Q, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.545809319 (1h16m2.32674271s) [noise=true , address=104.36.82.16:4293 , piece_id=D3PVF6AD6KLKK6V7TCXC7T4R3ILAZVJVCDREFPVG3SMYJTZYHIKQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.546515257 (1h16m2.327448646s) [noise=true , address=45.91.116.6:4974 , piece_id=66AHC6FRA3L6YS2OIHPAAUQDWNWPR3LXOMWZ2AVTY53QJXD4PIMA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.548171938 (1h16m2.329105319s) [noise=true , address=77.90.31.14:4663 , piece_id=MAHGERLTWI4DAP27TPQWJ3YRK3A2DA643QG5ZQ5B466SYGIV242A, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.548972775 (1h16m2.329906147s) [noise=true , address=104.36.82.9:4005 , piece_id=26LO3JCYB42R3QGC4ENR7XORRKUW76KHMWRWKOPO2AKPNUT2PAXQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.549556098 (1h16m2.330489479s) [noise=true , address=45.91.116.7:4010 , piece_id=P7GSPVLVGKS6YEMINKYIIA2E6MN3LH2CIMOT5MWTKYO2JJLMBNAA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.550145176 (1h16m2.331078566s) [noise=true , address=45.91.116.5:4299 , piece_id=J36A2QN72CO4ACOEEZXJ4BHYJAOQ533QYAK5QJC2N2M6ASZ6KJFQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.677350539 (1h16m2.458283921s) [upload=1175]: Next returned nil segment Jul 26 23:07:46.677455541 (1h16m2.458388927s) [upload=1175]: Waiting for error group managing segments... Jul 26 23:07:46.683203189 (1h16m2.464136574s) [noise=true , address=45.91.116.6:4974 , piece_id=66AHC6FRA3L6YS2OIHPAAUQDWNWPR3LXOMWZ2AVTY53QJXD4PIMA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684101593 (1h16m2.465034973s) [noise=true , address=45.91.116.5:4297 , piece_id=QOL343MXQK3D5PXHWAAXVJJXUVVRDSRXOYUGDXUWA6E6NFW5BW6A, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684245646 (1h16m2.465179013s) [noise=true , address=77.90.31.14:4663 , piece_id=MAHGERLTWI4DAP27TPQWJ3YRK3A2DA643QG5ZQ5B466SYGIV242A, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684707108 (1h16m2.465640489s) [noise=true , address=45.91.116.7:4010 , piece_id=P7GSPVLVGKS6YEMINKYIIA2E6MN3LH2CIMOT5MWTKYO2JJLMBNAA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684965874 (1h16m2.465899243s) [noise=true , address=45.91.116.8:4047 , piece_id=AVYWRB5JKMKRVUBM7CONKSFLWEVBMKOCHRV3ERSXNVTPB6VNFAEQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.686300892 (1h16m2.467234257s) [noise=true , address=45.91.116.5:4299 , piece_id=J36A2QN72CO4ACOEEZXJ4BHYJAOQ533QYAK5QJC2N2M6ASZ6KJFQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.687185916 (1h16m2.46811928s) [noise=true , address=45.91.116.4:4938 , piece_id=MVZ2K4WJEN4GNWCO2RXE6XCF7Y7YXWV5UVD6K4PPEIKUMK5JG7PQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:08:16.468753767 (1h16m32.249687156s) [noise=true , address=104.36.82.2:4729 , piece_id=XXZNOYSYRIIJIR54NTZXL7F7XXYIPCLJY4X6NGV2VLSD43UD2QFQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:38346->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468798397 (1h16m32.249731773s) [noise=true , address=104.36.82.9:4005 , piece_id=26LO3JCYB42R3QGC4ENR7XORRKUW76KHMWRWKOPO2AKPNUT2PAXQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:56398->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468859512 (1h16m32.249792889s) [noise=true , address=104.36.82.4:4802 , piece_id=WOP737BTCQ3G6NUWIGKRRZR6KYMPHN7WUJ5HOPZTNYHK45K77H7Q, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468923799 (1h16m32.249857169s) [noise=true , address=104.36.82.8:4970 , piece_id=WIZILE4HKCT4EBR7NO6USMQLVH3ANCRKXUIMDSRGK2W6BZNAOJGQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 159.223.143.39:39368->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468941115 (1h16m32.249874497s) [noise=true , address=104.36.82.15:4257 , piece_id=N7UGWSCY4LR5TPC5RED7YDIW5TIZMXPZ35SUWTVPIZTRR6BFMDMQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:36836->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468980078 (1h16m32.249913459s) [noise=true , address=104.36.82.15:4258 , piece_id=MSSR4RXTG4RQMJR6DWE2IISGHEHURQGIZUJ3TNZW5XX64ZGM3RLA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:52846->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469051468 (1h16m32.249984846s) [noise=true , address=104.36.82.16:4293 , piece_id=D3PVF6AD6KLKK6V7TCXC7T4R3ILAZVJVCDREFPVG3SMYJTZYHIKQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:38550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469067759 (1h16m32.250001137s) [noise=true , address=104.36.82.6:4875 , piece_id=MUBYCAFZZNFAWUZDOZZQIKAFQSIZWF7AEEZRF47INQ2EAF66NNZA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:48866->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469096507 (1h16m32.250029875s) [noise=true , address=104.36.82.14:4223 , piece_id=YQFOFJBGZMDVNGIAU7NR5HWMNN4SXWKKLNRFRYMMW2O6QUGA7SKQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:52262->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469153422 (1h16m32.250086957s) [seg_pos={0 0}, upload=1175]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:16.495582206 (1h16m32.276515679s) [upload=1175]: Done uploading segments... Jul 26 23:08:16.495628545 (1h16m32.276561947s) [upload=1175]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.032380819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X................X........................X......XX] duration: 30.033439715s FAIL (2025-07-26 23:08:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033490399s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001421453s FAIL (2025-07-26 23:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001448019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 218.912172ms FAIL (2025-07-26 23:07:44 UTC) orbiter warning: uplink: too many requests Duration: 218.944248ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XX] duration: 1m0.029265261s FAIL (2025-07-26 23:07:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== QTYDCF4B7ITENJ4IHP5CF2GQ5IBTVOA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.635602680 (1h14m56.416536068s) [noise=true , address=104.36.82.12:4182 , piece_id=7BUAGBJEH6U35GFWZJZ2ALOZDWJY6IHRTVUSQIH5P3WCZ57GFDWQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.635666958 (1h14m56.416600343s) [noise=true , address=104.36.82.4:4835 , piece_id=6ONX3EUIHN3ZV3ZXKLYPMMPPMNKDSGCPCR3I2SQZDJG6ODZIWVAA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636221817 (1h14m56.417155202s) [noise=true , address=79.127.205.234:4613 , piece_id=PS4KK6FV2GS6CA5ZUCG37XSWUWKJEXFO4IA6QQ4BBYTDUXIFJTSQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636282230 (1h14m56.417215617s) [noise=true , address=104.36.82.46:4513 , piece_id=3QXJWIYI2UCJSXY7DYYT36MLNLAKEZ7BFIXQ5PV4PXYG52UYTEAQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636628901 (1h14m56.417562281s) [noise=true , address=104.36.82.11:4139 , piece_id=VHTHZV5MFUKN5KNKLUQJMM7BZGBJCOALV6SFDWDKDQVSGDV72MJQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636837528 (1h14m56.417770913s) [noise=true , address=104.36.82.52:4091 , piece_id=A36364UTYE5BWAJLV7EBHMKEA7LXHEYSFPPKXBFGGLKQFRGEBL6Q, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637048027 (1h14m56.417981418s) [noise=true , address=104.36.82.56:4262 , piece_id=5P4NUWKQMZHUHAEDMS5RFI2B76DSBTVOCE7WUXUXK2YGUXZW47BA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637333667 (1h14m56.418267056s) [noise=true , address=79.127.205.235:4683 , piece_id=6YOBQICR5E5MDO3XDFHZSJK35Z7QTBGOYCKIJYV3JQTELC7FGI3A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637796008 (1h14m56.418729393s) [noise=true , address=104.36.82.9:4015 , piece_id=XE7TTRTSXBAQQI7G5TPTLXZ3DUOWMQ5JMI6M5R5EBJI76WKSLO2A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.638243205 (1h14m56.419176594s) [noise=true , address=104.36.82.22:4535 , piece_id=TIEDRQ7T5XR5RIY3YZE46JTWEIZJSNNIGTGGBFX5EY3FC2WU5I4A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.638639898 (1h14m56.41957327s) [noise=true , address=104.36.82.54:4172 , piece_id=T6PAR63ENE22FBS5OLL7E2MMVYLTJ3LUNFEAF36DRQKQSDAEPGKQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.822810191 (1h14m56.603743577s) [upload=1157]: Next returned nil segment Jul 26 23:06:40.822832993 (1h14m56.603766385s) [upload=1157]: Waiting for error group managing segments... Jul 26 23:06:40.853264663 (1h14m56.634198058s) [noise=true , address=79.127.220.101:4048 , piece_id=5WHWPJZSQ4JYUPXIEVTHDUGHBD3UGOIXPN7AC2VZOFTORRQPJC2Q, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:40.928131201 (1h14m56.709064575s) [noise=true , address=156.146.43.225:4449 , piece_id=LAUVUGX2Y73O4DKUK4DNR7O3H65SOK6LEGGNVKU32XND2BLX4AHQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:41.259976183 (1h14m57.040909555s) [noise=true , address=79.127.205.234:4613 , piece_id=PS4KK6FV2GS6CA5ZUCG37XSWUWKJEXFO4IA6QQ4BBYTDUXIFJTSQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:41.343659822 (1h14m57.12459319s) [noise=true , address=79.127.205.235:4683 , piece_id=6YOBQICR5E5MDO3XDFHZSJK35Z7QTBGOYCKIJYV3JQTELC7FGI3A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:07:36.063582207 (1h15m51.844515586s) [noise=true , address=104.36.82.56:4262 , piece_id=5P4NUWKQMZHUHAEDMS5RFI2B76DSBTVOCE7WUXUXK2YGUXZW47BA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:07:40.507730600 (1h15m56.288663977s) [noise=true , address=104.36.82.54:4172 , piece_id=T6PAR63ENE22FBS5OLL7E2MMVYLTJ3LUNFEAF36DRQKQSDAEPGKQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 159.223.143.39:44996->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.507836646 (1h15m56.288770016s) [noise=true , address=104.36.82.9:4015 , piece_id=XE7TTRTSXBAQQI7G5TPTLXZ3DUOWMQ5JMI6M5R5EBJI76WKSLO2A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: write tcp 159.223.143.39:54344->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.507935623 (1h15m56.288869008s) [noise=true , address=104.36.82.51:4049 , piece_id=PDJJ62LDJVUJXSIBRMK6LQXPSVRX7FCQLNAL6CNJWOVQLSKTMWGQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: write tcp 159.223.143.39:59346->104.36.82.51:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508198389 (1h15m56.289131752s) [noise=true , address=104.36.82.41:4771 , piece_id=2FLO4QF57SARKMTNRCURZMHLYRKGJOMMHW6CBAW2QWCJM7QMOFLA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508431420 (1h15m56.289364802s) [noise=true , address=104.36.82.11:4139 , piece_id=VHTHZV5MFUKN5KNKLUQJMM7BZGBJCOALV6SFDWDKDQVSGDV72MJQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: rpc: dial tcp 104.36.82.11:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508498866 (1h15m56.289432229s) [noise=true , address=104.36.82.46:4513 , piece_id=3QXJWIYI2UCJSXY7DYYT36MLNLAKEZ7BFIXQ5PV4PXYG52UYTEAQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: read tcp 159.223.143.39:37300->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508592196 (1h15m56.289525573s) [noise=true , address=104.36.82.47:4517 , piece_id=PPDGZ2NUDEJEJNTHMETWVQAY4RRLFUUTUAYETXOPKHNYSVZRWJGA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508661526 (1h15m56.289594903s) [noise=true , address=104.36.82.49:4978 , piece_id=FDORIP3PHQQXXEXGWTX6Y4MWCOSFFNPTK24ZQX5QQA5ANGYGFX6A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 159.223.143.39:53376->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508743353 (1h15m56.289676732s) [noise=true , address=104.36.82.12:4182 , piece_id=7BUAGBJEH6U35GFWZJZ2ALOZDWJY6IHRTVUSQIH5P3WCZ57GFDWQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1KSPtVhJPKYAb3Gj25ihyExNgkDCGo9QTDaFcEfXRfLdDz8nsh, address:104.36.82.12:4182): protocol: write tcp 159.223.143.39:43074->104.36.82.12:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508779785 (1h15m56.289713162s) [noise=true , address=104.36.82.22:4535 , piece_id=TIEDRQ7T5XR5RIY3YZE46JTWEIZJSNNIGTGGBFX5EY3FC2WU5I4A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 159.223.143.39:50904->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508918006 (1h15m56.289851394s) [noise=true , address=104.36.82.15:4278 , piece_id=PRNJLZPCB67YIVT3HXXTAQTYDCF4B7ITENJ4IHP5CF2GQ5IBTVOA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12qpbyKnJ8qXmos7gdxd46F8QXx98LUwtKD6mryaXPBDD5WFSwu, address:104.36.82.15:4278): protocol: rpc: dial tcp 104.36.82.15:4278: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508993576 (1h15m56.289926958s) [noise=true , address=104.36.82.4:4835 , piece_id=6ONX3EUIHN3ZV3ZXKLYPMMPPMNKDSGCPCR3I2SQZDJG6ODZIWVAA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:104.36.82.4:4835): protocol: rpc: dial tcp 104.36.82.4:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.509012114 (1h15m56.289945491s) [noise=true , address=104.36.82.52:4091 , piece_id=A36364UTYE5BWAJLV7EBHMKEA7LXHEYSFPPKXBFGGLKQFRGEBL6Q, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: write tcp 159.223.143.39:41692->104.36.82.52:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.509054458 (1h15m56.289987826s) [seg_pos={0 0}, upload=1157]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 23:07:40.534043629 (1h15m56.314977014s) [upload=1157]: Done uploading segments... Jul 26 23:07:40.534076653 (1h15m56.315010056s) [upload=1157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029295627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X................X........................X......X] duration: 30.07114153s FAIL (2025-07-26 23:07:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.071174977s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 334.632677ms FAIL (2025-07-26 23:06:56 UTC) orbiter warning: uplink: too many requests Duration: 334.659199ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.....X.X.X...XX.................X.X..........XX...X...X..XX.] duration: 1.851735979s FAIL (2025-07-26 23:06:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.851778442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......X.X.X...XX.................X.X..........XX...X...X..XX] duration: 1m0.000637893s FAIL (2025-07-26 23:06:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000660082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 760.79387ms FAIL (2025-07-26 23:06:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 760.936161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.X] duration: 1m0.051609237s FAIL (2025-07-26 23:06:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== oad=1141]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:05:28.628594097 (1h13m44.409527477s) [seg_pos={0 0}, upload=1141]: Begin upload segment... Jul 26 23:05:28.628694608 (1h13m44.409627996s) [noise=true , address=77.90.31.11:4565 , piece_id=3LZX5FCWTPBPO626BNXBHOLLBXX2DTQ5WBMPE6HBOAZZLNZOUXKA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.629244531 (1h13m44.41017792s) [seg_pos={0 0}, upload=1141]: Done begin upload segment. Jul 26 23:05:28.629283339 (1h13m44.410216725s) [noise=true , address=156.146.43.225:4449 , piece_id=UU3IT2XIESCSKQBDSBWRAZL4TAU4JYVXD3FVHPFCAZ7LI4FM2BQA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.629359950 (1h13m44.410293327s) [noise=true , address=79.127.219.47:4048 , piece_id=MOYSRZRTPDXDIL7OIR2AP72WPD6462GKBQ2NOWU364EW3FW3TK4A, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.631629535 (1h13m44.412562929s) [noise=true , address=45.91.116.11:4151 , piece_id=SAYXV67WSPMUNBXBAYEUBU2KDEVJ6Z7DSWGOTETLC65Q47JWPMOA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.631877702 (1h13m44.412811085s) [noise=true , address=79.127.219.38:4017 , piece_id=ULNTEQB7GSZXRVQ4J2H6VYAQNUFP6UNSOBXL3TKWOHWN5R5QF5ZA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.633078788 (1h13m44.414012177s) [noise=true , address=45.91.116.14:4206 , piece_id=FSRAFGVLLUL6K7GCMQSUTLN2CPDHYZY6SLV45A72CI27VF4IFOPQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.637171961 (1h13m44.418105351s) [noise=true , address=79.127.205.245:4223 , piece_id=I64JI4ALYKNI6AZYSHKEXUS3ZRMEUA52ZOHUZ7LZI6SYP7GFJECQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.637820761 (1h13m44.41875415s) [noise=true , address=79.127.219.51:4210 , piece_id=RP7FUKX3U33FZAMWG72S2N3OV4NWZRP3TBR3CIPE7FNXQ3CEPXVA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.638393389 (1h13m44.419326781s) [noise=true , address=77.90.31.11:4587 , piece_id=4S5GNVLB24GSNOTFW6TZSZIBPLCLFPOVFBBLTV4N36WIXQGQGF7Q, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.639013958 (1h13m44.419947347s) [noise=true , address=45.91.116.5:4310 , piece_id=NOWAW7ZMMBYPZ2ZPRHQ3BWRIMTR7ETC6GAOQJT6GCWLZ6HQREEKA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.643890927 (1h13m44.424824315s) [noise=true , address=77.90.31.17:4774 , piece_id=77GYDBWYEOEEPU53D6OQCWIEOXPJ56O7MH3BLUPM4ABFDOQNU5XQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.644616235 (1h13m44.425549625s) [noise=true , address=79.127.201.214:4321 , piece_id=SFQ2776UVZOKYK3H6A5D3RLHL2VMAWD2S24REJSBIUH4WJFEI4JQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.645154693 (1h13m44.426088079s) [noise=true , address=64.49.13.138:4482 , piece_id=XN4FDKAFB5NZ3P2VUXJVEMT3QRLJMPZW6YBI7C4KYFXC22MPCESQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.645620447 (1h13m44.426553836s) [noise=true , address=156.146.43.225:4434 , piece_id=I26LVHIPQATHFFGEFKRB3DFSFX4SFE6RSXCVOMEIRDEGMOIYOVWA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646080005 (1h13m44.427013392s) [noise=true , address=45.91.116.14:4222 , piece_id=O3B6MN4QCCMDV7XBYGUB2JW2SPCZ7FTX7SNKJKVWLFX4TXSR7QFQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646528660 (1h13m44.427462044s) [noise=true , address=45.91.116.9:4277 , piece_id=C4NTD6SJIAZ6SKVUOXZYCU3B6TADZXTBZUIXBTDQLALKRF6MZBFA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646938102 (1h13m44.427871491s) [noise=true , address=64.49.13.139:4478 , piece_id=X54V5YMKRWULPYSZTYJ4EGEHZFGVZ5SRQHTTQW56PLSNHGB7HBWQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.647785833 (1h13m44.42871922s) [noise=true , address=79.127.205.234:4614 , piece_id=HOACXUX656IWAZXPGSMH6EQQJB7FZ3NK4MJSLW5M62MCKLBGVA5Q, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.710076062 (1h13m44.491009442s) [noise=true , address=79.127.219.47:4048 , piece_id=MOYSRZRTPDXDIL7OIR2AP72WPD6462GKBQ2NOWU364EW3FW3TK4A, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.739826655 (1h13m44.520760032s) [upload=1141]: Next returned nil segment Jul 26 23:05:28.739849485 (1h13m44.520782852s) [upload=1141]: Waiting for error group managing segments... Jul 26 23:05:28.810592579 (1h13m44.591525978s) [noise=true , address=156.146.43.225:4434 , piece_id=I26LVHIPQATHFFGEFKRB3DFSFX4SFE6RSXCVOMEIRDEGMOIYOVWA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.848581660 (1h13m44.629515041s) [noise=true , address=79.127.219.51:4210 , piece_id=RP7FUKX3U33FZAMWG72S2N3OV4NWZRP3TBR3CIPE7FNXQ3CEPXVA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.997405787 (1h13m44.778339151s) [noise=true , address=79.127.219.38:4017 , piece_id=ULNTEQB7GSZXRVQ4J2H6VYAQNUFP6UNSOBXL3TKWOHWN5R5QF5ZA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.269908065 (1h13m45.050841438s) [noise=true , address=79.127.205.234:4614 , piece_id=HOACXUX656IWAZXPGSMH6EQQJB7FZ3NK4MJSLW5M62MCKLBGVA5Q, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.327078430 (1h13m45.108011811s) [noise=true , address=79.127.205.245:4223 , piece_id=I64JI4ALYKNI6AZYSHKEXUS3ZRMEUA52ZOHUZ7LZI6SYP7GFJECQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.740795414 (1h13m45.521728784s) [noise=true , address=64.49.13.139:4478 , piece_id=X54V5YMKRWULPYSZTYJ4EGEHZFGVZ5SRQHTTQW56PLSNHGB7HBWQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:30.854867267 (1h13m46.635800653s) [noise=true , address=79.127.201.214:4321 , piece_id=SFQ2776UVZOKYK3H6A5D3RLHL2VMAWD2S24REJSBIUH4WJFEI4JQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:32.001768315 (1h13m47.782701697s) [noise=true , address=156.146.43.225:4449 , piece_id=UU3IT2XIESCSKQBDSBWRAZL4TAU4JYVXD3FVHPFCAZ7LI4FM2BQA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:32.710987582 (1h13m48.491920961s) [noise=true , address=64.49.13.138:4482 , piece_id=XN4FDKAFB5NZ3P2VUXJVEMT3QRLJMPZW6YBI7C4KYFXC22MPCESQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:37.131083709 (1h13m52.912017078s) [noise=true , address=45.91.116.11:4151 , piece_id=SAYXV67WSPMUNBXBAYEUBU2KDEVJ6Z7DSWGOTETLC65Q47JWPMOA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:39.069558528 (1h13m54.850491928s) [noise=true , address=45.91.116.9:4277 , piece_id=C4NTD6SJIAZ6SKVUOXZYCU3B6TADZXTBZUIXBTDQLALKRF6MZBFA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:39.211670250 (1h13m54.992603615s) [noise=true , address=45.91.116.14:4222 , piece_id=O3B6MN4QCCMDV7XBYGUB2JW2SPCZ7FTX7SNKJKVWLFX4TXSR7QFQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:40.339113657 (1h13m56.12004703s) [noise=true , address=45.91.116.5:4310 , piece_id=NOWAW7ZMMBYPZ2ZPRHQ3BWRIMTR7ETC6GAOQJT6GCWLZ6HQREEKA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:06:09.158854012 (1h14m24.939787384s) [noise=true , address=77.90.31.17:4774 , piece_id=77GYDBWYEOEEPU53D6OQCWIEOXPJ56O7MH3BLUPM4ABFDOQNU5XQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:06:15.253853134 (1h14m31.034786518s) [noise=true , address=45.91.116.14:4206 , piece_id=FSRAFGVLLUL6K7GCMQSUTLN2CPDHYZY6SLV45A72CI27VF4IFOPQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:06:28.443449991 (1h14m44.224383381s) [noise=true , address=77.90.31.11:4587 , piece_id=4S5GNVLB24GSNOTFW6TZSZIBPLCLFPOVFBBLTV4N36WIXQGQGF7Q, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: write tcp 159.223.143.39:41962->77.90.31.11:4587: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:28.443565474 (1h14m44.224498859s) [noise=true , address=77.90.31.11:4565 , piece_id=3LZX5FCWTPBPO626BNXBHOLLBXX2DTQ5WBMPE6HBOAZZLNZOUXKA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: rpc: dial tcp 77.90.31.11:4565: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:28.443604917 (1h14m44.224538297s) [seg_pos={0 0}, upload=1141]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:06:28.492699374 (1h14m44.273632744s) [upload=1141]: Done uploading segments... Jul 26 23:06:28.492731778 (1h14m44.273665147s) [upload=1141]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 1m0.051632624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X................X........................X......] duration: 1.068886634s FAIL (2025-07-26 23:05:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.068921703s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 753.77877ms FAIL (2025-07-26 23:05:44 UTC) orbiter warning: uplink: too many requests Duration: 753.812407ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......X.X.X...XX.................X.X..........XX...X...X..X] duration: 1m0.000890462s FAIL (2025-07-26 23:05:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000938371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 428.613764ms FAIL (2025-07-26 23:05:29 UTC) orbiter warning: uplink: too many requests Duration: 428.641952ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.] duration: 17.874108114s FAIL (2025-07-26 23:05:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6 23:04:10.417101156 (1h12m26.198034544s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417726595 (1h12m26.198659981s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417914592 (1h12m26.198847979s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418823898 (1h12m26.199757285s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418996906 (1h12m26.199930268s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.419970568 (1h12m26.200903957s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420440076 (1h12m26.201373465s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420989928 (1h12m26.201923315s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421356482 (1h12m26.202289873s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421716261 (1h12m26.202649651s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422228566 (1h12m26.203161955s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422867505 (1h12m26.203800898s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423282554 (1h12m26.204215936s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423768254 (1h12m26.204701641s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.424451766 (1h12m26.20538513s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.431524809 (1h12m26.212458194s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.433610399 (1h12m26.214543773s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.435563240 (1h12m26.216496606s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.463289025 (1h12m26.244222393s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.500486501 (1h12m26.28141989s) [upload=1121]: Next returned nil segment Jul 26 23:04:10.500511419 (1h12m26.28144482s) [upload=1121]: Waiting for error group managing segments... Jul 26 23:04:10.513835566 (1h12m26.294768931s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:10.585488550 (1h12m26.366421933s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:13Jtv8kL5R9oVro2D4ioqsznoJBXyNgJLdj1FcrGN3WWH45rkP, address:79.127.201.215:4354): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.593364447 (1h12m26.374297834s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:11.029127822 (1h12m26.810061191s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.030089513 (1h12m26.81102289s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.039837172 (1h12m26.820770535s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.049753426 (1h12m26.830686806s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:12.300355942 (1h12m28.081289319s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:12.337009638 (1h12m28.117943032s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.175933117 (1h12m28.956866501s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.280233882 (1h12m29.061167268s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:14.871557683 (1h12m30.652491078s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:18.580092593 (1h12m34.361025956s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:05:10.332106674 (1h13m26.113040057s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:05:10.332198466 (1h13m26.113131843s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 159.223.143.39:49606->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:10.332304942 (1h13m26.113238319s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332387108 (1h13m26.113320485s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332414088 (1h13m26.113347476s) [seg_pos={0 0}, upload=1121]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:05:10.417384543 (1h13m26.19831792s) [upload=1121]: Done uploading segments... Jul 26 23:05:10.417409881 (1h13m26.19834325s) [upload=1121]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 17.874133346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X................X........................X.....] duration: 672.995755ms FAIL (2025-07-26 23:05:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 673.087823ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........X..................................................] duration: 768.238468ms FAIL (2025-07-26 23:05:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 768.271042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X] duration: 1m0.087427744s FAIL (2025-07-26 23:05:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6 23:04:10.417101156 (1h12m26.198034544s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417726595 (1h12m26.198659981s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417914592 (1h12m26.198847979s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418823898 (1h12m26.199757285s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418996906 (1h12m26.199930268s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.419970568 (1h12m26.200903957s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420440076 (1h12m26.201373465s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420989928 (1h12m26.201923315s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421356482 (1h12m26.202289873s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421716261 (1h12m26.202649651s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422228566 (1h12m26.203161955s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422867505 (1h12m26.203800898s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423282554 (1h12m26.204215936s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423768254 (1h12m26.204701641s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.424451766 (1h12m26.20538513s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.431524809 (1h12m26.212458194s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.433610399 (1h12m26.214543773s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.435563240 (1h12m26.216496606s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.463289025 (1h12m26.244222393s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.500486501 (1h12m26.28141989s) [upload=1121]: Next returned nil segment Jul 26 23:04:10.500511419 (1h12m26.28144482s) [upload=1121]: Waiting for error group managing segments... Jul 26 23:04:10.513835566 (1h12m26.294768931s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:10.585488550 (1h12m26.366421933s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:13Jtv8kL5R9oVro2D4ioqsznoJBXyNgJLdj1FcrGN3WWH45rkP, address:79.127.201.215:4354): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.593364447 (1h12m26.374297834s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:11.029127822 (1h12m26.810061191s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.030089513 (1h12m26.81102289s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.039837172 (1h12m26.820770535s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.049753426 (1h12m26.830686806s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:12.300355942 (1h12m28.081289319s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:12.337009638 (1h12m28.117943032s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.175933117 (1h12m28.956866501s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.280233882 (1h12m29.061167268s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:14.871557683 (1h12m30.652491078s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:18.580092593 (1h12m34.361025956s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:05:10.332106674 (1h13m26.113040057s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:05:10.332198466 (1h13m26.113131843s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 159.223.143.39:49606->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:10.332304942 (1h13m26.113238319s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332387108 (1h13m26.113320485s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332414088 (1h13m26.113347476s) [seg_pos={0 0}, upload=1121]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:05:10.417384543 (1h13m26.19831792s) [upload=1121]: Done uploading segments... Jul 26 23:05:10.417409881 (1h13m26.19834325s) [upload=1121]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.087458277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X................X........................X....] duration: 636.182191ms FAIL (2025-07-26 23:04:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 636.252781ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........X.................................................] duration: 462.208902ms FAIL (2025-07-26 23:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.238368ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 197.952963ms FAIL (2025-07-26 23:04:16 UTC) orbiter warning: uplink: too many requests Duration: 197.986264ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X................X........................X...] duration: 670.712401ms FAIL (2025-07-26 23:03:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 670.744791ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.] duration: 11.38683419s FAIL (2025-07-26 23:03:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 23:02:38.009431979 (1h10m53.790365367s) [upload=1100]: Starting upload Jul 26 23:02:38.009529742 (1h10m53.79046312s) [upload=1100]: Done starting upload Jul 26 23:02:38.009693140 (1h10m53.790626551s) [upload=1100]: Uploading segments... Jul 26 23:02:38.009988782 (1h10m53.790922148s) [upload=1100]: Got next segment. Inline: false Jul 26 23:02:38.010070015 (1h10m53.7910034s) [upload=1100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:02:38.010077679 (1h10m53.791011064s) [upload=1100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:02:38.044231887 (1h10m53.825165274s) [seg_pos={0 0}, upload=1100]: Begin upload segment... Jul 26 23:02:38.044288604 (1h10m53.825221986s) [seg_pos={0 0}, upload=1100]: Done begin upload segment. Jul 26 23:02:38.044317043 (1h10m53.82525043s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044360663 (1h10m53.825294057s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044838801 (1h10m53.825772189s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044953426 (1h10m53.82588681s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045307782 (1h10m53.826241184s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045577917 (1h10m53.826511306s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045795589 (1h10m53.826728975s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046160990 (1h10m53.827094381s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046168935 (1h10m53.827102298s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046544662 (1h10m53.827478048s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046638748 (1h10m53.82757213s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046971631 (1h10m53.827905015s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047469479 (1h10m53.828402863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047886394 (1h10m53.828819784s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048312600 (1h10m53.829245988s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048741550 (1h10m53.829674942s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048867751 (1h10m53.829801138s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.049276904 (1h10m53.830210291s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.113183749 (1h10m53.894117126s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:38.135417853 (1h10m53.916351218s) [upload=1100]: Next returned nil segment Jul 26 23:02:38.135441335 (1h10m53.916374722s) [upload=1100]: Waiting for error group managing segments... Jul 26 23:02:38.296608477 (1h10m54.077541863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.553667152 (1h10m54.334600522s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.631703695 (1h10m54.412637075s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.646996395 (1h10m54.427929767s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.660623879 (1h10m54.441557271s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.661444603 (1h10m54.442377975s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.480167315 (1h10m56.261100702s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.846007988 (1h10m56.626941368s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.905994229 (1h10m56.686927607s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:44.563162762 (1h11m0.344096151s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:45.387863794 (1h11m1.168797163s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:46.641102649 (1h11m2.422036011s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:56.398068001 (1h11m12.179001368s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:12.478927638 (1h11m28.259861015s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:22.198648346 (1h11m37.979581727s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:37.964568820 (1h11m53.7455022s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 159.223.143.39:42210->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.965835006 (1h11m53.746768388s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 159.223.143.39:36482->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.967314376 (1h11m53.74824777s) [seg_pos={0 0}, upload=1100]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:03:37.994728808 (1h11m53.775662191s) [upload=1100]: Done uploading segments... Jul 26 23:03:37.994772690 (1h11m53.775706073s) [upload=1100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.386861202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [........X.X.X...XX.................X.X..........XX...X...X..] duration: 13.316063426s FAIL (2025-07-26 23:03:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.316242504s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)