Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX.] duration: 525.713923ms FAIL (2025-07-26 23:59:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 525.765731ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X...X..X.......X...X....XX..............X.X........X...XX...] duration: 40.256455503s FAIL (2025-07-26 23:59:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.256480071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.X.......XXX...XX.....X.XXX...............X..X......X.X] duration: 30.002146528s FAIL (2025-07-26 23:58:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002172781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX] duration: 30.050569176s FAIL (2025-07-26 23:58:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.05059587s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.036569758s FAIL (2025-07-26 23:58:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== begin upload segment. Jul 26 23:57:51.957513831 (2h6m7.738447215s) [noise=true , address=104.36.82.52:4102 , piece_id=3RMUU2DRTI7IGVSPNBGYSNBDGFO5DK5GOMU25JTSWMPU6PURZG4Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.957545107 (2h6m7.738478493s) [noise=true , address=104.36.82.55:4225 , piece_id=DZWY5PIEPQHYRRLNYB5GHYTWPCSFHKCK2R5EDNWWUWSNBYUNOSWA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.957924627 (2h6m7.738858005s) [noise=true , address=79.127.205.235:4679 , piece_id=GO3AKU6756JZJVJOYZ63HCUIRLAJ7WGDW4UBMO7GI7R4F7H3WYPQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958013352 (2h6m7.738946734s) [noise=true , address=77.90.31.16:4746 , piece_id=SSN2EODN36LBAOVGSJYE2ITWKFJEKLJHL4RCCM5XKEQBSAWKD6MQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958277241 (2h6m7.739210619s) [noise=true , address=77.90.31.13:4644 , piece_id=CW6IFOE2IA6AYIKOX6PCA7JTUFHTB4GFHFZADMASAEGE7BHE54LA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958415549 (2h6m7.739348933s) [noise=true , address=104.36.82.38:4700 , piece_id=UOQG7EW6VWSVCVT2BJRWF4KDL6FLMPNZB5QUGBXQP73BHYWZLM2A, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958568217 (2h6m7.739501594s) [noise=true , address=104.36.82.53:4141 , piece_id=2DRWXXQIRVWGAGYKTQ7H27WASIKEOD7XRRVPVI7BBGKZRKM4FV6Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958946407 (2h6m7.739879788s) [noise=true , address=77.90.31.11:4578 , piece_id=4EOHNPI2SFNZD3OKWAG6YCAX4NATOP5AIIBOLIATDFYR5LAXDDMA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958966345 (2h6m7.739899733s) [noise=true , address=64.49.13.140:4508 , piece_id=SSNJICPKVEUDZYMOY5AYLTI7ITU6IXZ3OUAKY7FMGV753EPZTK3Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959349291 (2h6m7.740282667s) [noise=true , address=104.36.82.54:4177 , piece_id=7VJMYUHPL52XQV7AMLP2TGKGNJDLOPA5E3PRWCCAD6I6DPY43UPQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959371783 (2h6m7.740305168s) [noise=true , address=104.36.82.56:4237 , piece_id=G2V3S5LU4R5J573RG6FA3V6HTHUNO4D2EP6FVKF2LQIMADS3N6ZQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959671337 (2h6m7.740604715s) [noise=true , address=77.90.31.15:4703 , piece_id=W6OWZDJBWXXFVUE6FUP6RYLTQABXOZAKYL335XOQQ2SJFIVFVQCA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959680516 (2h6m7.740613897s) [noise=true , address=104.36.82.46:4510 , piece_id=BZ73KMXSX54TGAIC74JYOBAPMCFD732CNQ2F4BHPUKOWOEQVFG7A, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959937415 (2h6m7.740870791s) [noise=true , address=79.127.219.38:4010 , piece_id=WC566RDAZR2JPSLDOQ2JKFJ7TCCHXTZB26BVPMI6VMXMJNMG65BA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960255168 (2h6m7.741188554s) [noise=true , address=104.36.82.47:4532 , piece_id=N5II752EGQ6IDVNF6VPETZGNJODGRM4425KBCDM47YKTR2XUDDGA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960346598 (2h6m7.741279967s) [noise=true , address=79.127.201.214:4321 , piece_id=W4NHV25OXI3WQEV3JB6VOEZBPULGUSHRLYTGELLVK7UWQJW6IGTQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960730290 (2h6m7.741663655s) [noise=true , address=64.49.13.137:4502 , piece_id=O3OMBCB6YSEWDRRASM7JGOLHFTOUTKLGJ5JLX2EWUQP472HIOSRA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.961176005 (2h6m7.742109383s) [noise=true , address=79.127.205.246:4279 , piece_id=K2WS2TP4WHV7VMHZS4CJMBRGO2PMCDUXQ6WJXQBGEFONDAPU2YGQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:52.040596586 (2h6m7.821529955s) [upload=1935]: Next returned nil segment Jul 26 23:57:52.040614900 (2h6m7.821548282s) [upload=1935]: Waiting for error group managing segments... Jul 26 23:57:52.079489604 (2h6m7.860422972s) [noise=true , address=77.90.31.16:4746 , piece_id=SSN2EODN36LBAOVGSJYE2ITWKFJEKLJHL4RCCM5XKEQBSAWKD6MQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.079882730 (2h6m7.860816109s) [noise=true , address=77.90.31.11:4578 , piece_id=4EOHNPI2SFNZD3OKWAG6YCAX4NATOP5AIIBOLIATDFYR5LAXDDMA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.417872880 (2h6m8.198806256s) [noise=true , address=79.127.219.38:4010 , piece_id=WC566RDAZR2JPSLDOQ2JKFJ7TCCHXTZB26BVPMI6VMXMJNMG65BA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.531510614 (2h6m8.312443982s) [noise=true , address=79.127.205.235:4679 , piece_id=GO3AKU6756JZJVJOYZ63HCUIRLAJ7WGDW4UBMO7GI7R4F7H3WYPQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.661261963 (2h6m8.442195343s) [noise=true , address=79.127.205.246:4279 , piece_id=K2WS2TP4WHV7VMHZS4CJMBRGO2PMCDUXQ6WJXQBGEFONDAPU2YGQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.887288970 (2h6m8.66822235s) [noise=true , address=64.49.13.137:4502 , piece_id=O3OMBCB6YSEWDRRASM7JGOLHFTOUTKLGJ5JLX2EWUQP472HIOSRA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:54.986003119 (2h6m10.766936496s) [noise=true , address=77.90.31.13:4644 , piece_id=CW6IFOE2IA6AYIKOX6PCA7JTUFHTB4GFHFZADMASAEGE7BHE54LA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:55.307965874 (2h6m11.088899252s) [noise=true , address=79.127.201.214:4321 , piece_id=W4NHV25OXI3WQEV3JB6VOEZBPULGUSHRLYTGELLVK7UWQJW6IGTQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:57.817915300 (2h6m13.598848678s) [noise=true , address=64.49.13.140:4508 , piece_id=SSNJICPKVEUDZYMOY5AYLTI7ITU6IXZ3OUAKY7FMGV753EPZTK3Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:58:19.402176971 (2h6m35.183110337s) [noise=true , address=77.90.31.15:4703 , piece_id=W6OWZDJBWXXFVUE6FUP6RYLTQABXOZAKYL335XOQQ2SJFIVFVQCA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:58:51.894658270 (2h7m7.675591646s) [noise=true , address=104.36.82.52:4102 , piece_id=3RMUU2DRTI7IGVSPNBGYSNBDGFO5DK5GOMU25JTSWMPU6PURZG4Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: write tcp 159.223.143.39:35084->104.36.82.52:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.894738332 (2h7m7.675671707s) [noise=true , address=104.36.82.47:4532 , piece_id=N5II752EGQ6IDVNF6VPETZGNJODGRM4425KBCDM47YKTR2XUDDGA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.894809469 (2h7m7.675742844s) [noise=true , address=104.36.82.55:4225 , piece_id=DZWY5PIEPQHYRRLNYB5GHYTWPCSFHKCK2R5EDNWWUWSNBYUNOSWA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:51.894949495 (2h7m7.675882871s) [noise=true , address=104.36.82.38:4700 , piece_id=UOQG7EW6VWSVCVT2BJRWF4KDL6FLMPNZB5QUGBXQP73BHYWZLM2A, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12JmsGbYEHsP6Pf2HinsbdsLZW85jnaHe8gHZXHcShCTxMxXJKP, address:104.36.82.38:4700): protocol: rpc: dial tcp 104.36.82.38:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895012990 (2h7m7.675946365s) [noise=true , address=104.36.82.56:4237 , piece_id=G2V3S5LU4R5J573RG6FA3V6HTHUNO4D2EP6FVKF2LQIMADS3N6ZQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 159.223.143.39:55096->104.36.82.56:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895065796 (2h7m7.67599917s) [noise=true , address=104.36.82.54:4177 , piece_id=7VJMYUHPL52XQV7AMLP2TGKGNJDLOPA5E3PRWCCAD6I6DPY43UPQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:40814->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895102232 (2h7m7.676035607s) [noise=true , address=104.36.82.46:4510 , piece_id=BZ73KMXSX54TGAIC74JYOBAPMCFD732CNQ2F4BHPUKOWOEQVFG7A, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:35032->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895141702 (2h7m7.67607508s) [noise=true , address=104.36.82.53:4141 , piece_id=2DRWXXQIRVWGAGYKTQ7H27WASIKEOD7XRRVPVI7BBGKZRKM4FV6Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: write tcp 159.223.143.39:44090->104.36.82.53:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895170857 (2h7m7.676104242s) [seg_pos={0 0}, upload=1935]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:58:51.928281324 (2h7m7.709214702s) [upload=1935]: Done uploading segments... Jul 26 23:58:51.928312614 (2h7m7.709245991s) [upload=1935]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.036613145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.029916228s FAIL (2025-07-26 23:58:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== : write tcp 159.223.143.39:36770->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538990054 (2h6m6.319923431s) [seg_pos={0 0}, upload=1927]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:57:50.564064323 (2h6m6.344997692s) [upload=1927]: Done uploading segments... Jul 26 23:57:50.564195664 (2h6m6.345129035s) [upload=1927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:57:50.669282712 (2h6m6.450216098s) [upload=1933]: Starting upload Jul 26 23:57:50.669376244 (2h6m6.450309628s) [upload=1933]: Done starting upload Jul 26 23:57:50.669594624 (2h6m6.450527991s) [upload=1933]: Uploading segments... Jul 26 23:57:50.669799987 (2h6m6.450733366s) [upload=1933]: Got next segment. Inline: false Jul 26 23:57:50.669871688 (2h6m6.450805071s) [upload=1933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:50.669877758 (2h6m6.450811137s) [upload=1933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:50.699301330 (2h6m6.480234711s) [seg_pos={0 0}, upload=1933]: Begin upload segment... Jul 26 23:57:50.699350004 (2h6m6.48028337s) [seg_pos={0 0}, upload=1933]: Done begin upload segment. Jul 26 23:57:50.699375683 (2h6m6.480309061s) [noise=true , address=45.91.116.13:4155 , piece_id=ZDC4R62AE7ONI56MW3SKTSXEE2XHI3DZ54ZTEED4IATQ3YRYBPNQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699414688 (2h6m6.480348073s) [noise=true , address=77.90.31.18:4805 , piece_id=OU5XZPBHVNCWSY3FEH5OAYETFDPZQADQGD6OJC2G7QZZEJJEDW4Q, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699761291 (2h6m6.480694669s) [noise=true , address=104.36.82.2:4732 , piece_id=JIPLHZ2S6XEHNR35KRHAPTX6DNLK7TFWKI4X5L5FNQUELJY3N3FQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699884427 (2h6m6.480817947s) [noise=true , address=45.91.116.4:4938 , piece_id=UQCFID7ANH3C62D6JWY6SXDRNO5DHNI72CNTZ223KYJ643DHLRSQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700300544 (2h6m6.481233925s) [noise=true , address=104.36.82.11:4113 , piece_id=LW2LDEUEPF4RM6JQZMJBXAATTUNQQPN42HH4YDJNR3AE5BYZFE4Q, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700453668 (2h6m6.481387053s) [noise=true , address=77.90.31.14:4661 , piece_id=XXOGW6JROE4IQSE2B6IZAVN3D6J3IW4HAHWOSUFTBBQ3QB6XJUBQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700693612 (2h6m6.481626997s) [noise=true , address=45.91.116.14:4190 , piece_id=FYVCED2JOP452LAH74LD2G2X4QYK4BQKBQ6E46W5FBWCLIYOXYGQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700866568 (2h6m6.481799946s) [noise=true , address=104.36.82.8:4969 , piece_id=62UXYO4GV3TZFAFNR6XGFQJJ67PI4MRS76GQNHQRZVUCE4UVPFRA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701036720 (2h6m6.481970103s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701277262 (2h6m6.482210643s) [noise=true , address=45.91.116.10:4083 , piece_id=4F77NBEDGF4CH45OI3DU5F32NNFNJFJFIJTG6MSM7PMOMASCID2A, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701394578 (2h6m6.482327957s) [noise=true , address=45.91.116.6:4974 , piece_id=H4FLJYQAOKMYZPDUHEHU55AOWUKK7MSQVSY5OSVKYF72KWQCEXXA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701719862 (2h6m6.48265326s) [noise=true , address=77.90.31.15:4698 , piece_id=T4YSEM442SNZIQOUPQ2A4XMAJ5AQK6O3A5ID5UNDI6ZENRRDPF7A, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702107535 (2h6m6.483040922s) [noise=true , address=77.90.31.15:4699 , piece_id=UWIGVBJFTBTX7LOAUMZFTVG7ZNNIWC5X62BNRKTUY6337QZF4YGA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702489223 (2h6m6.48342261s) [noise=true , address=104.36.82.13:4187 , piece_id=SMJMVDB7S47W3IDN2SU7CI3OLTNUZAGKRSVNPVFBIBZHZHWEPJIQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702861497 (2h6m6.483794878s) [noise=true , address=104.36.82.12:4150 , piece_id=J3XITNETGNAB7LPL5PPNY2VFJPWAQDMTDNN4DWXPZSN2LR6IJ3LA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.703183106 (2h6m6.484116484s) [noise=true , address=104.36.82.4:4802 , piece_id=FJBX5ZZNJDPTWKXS6TJDEDFEEIL7HJNRDXHRREKWGWASDYAHAOGQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.779050485 (2h6m6.559983854s) [upload=1933]: Next returned nil segment Jul 26 23:57:50.779066732 (2h6m6.560000107s) [upload=1933]: Waiting for error group managing segments... Jul 26 23:57:50.783242486 (2h6m6.564175854s) [noise=true , address=45.91.116.13:4155 , piece_id=ZDC4R62AE7ONI56MW3SKTSXEE2XHI3DZ54ZTEED4IATQ3YRYBPNQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.783857133 (2h6m6.564790501s) [noise=true , address=45.91.116.6:4974 , piece_id=H4FLJYQAOKMYZPDUHEHU55AOWUKK7MSQVSY5OSVKYF72KWQCEXXA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.784490796 (2h6m6.565424173s) [noise=true , address=45.91.116.4:4938 , piece_id=UQCFID7ANH3C62D6JWY6SXDRNO5DHNI72CNTZ223KYJ643DHLRSQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.784907248 (2h6m6.565840613s) [noise=true , address=77.90.31.14:4661 , piece_id=XXOGW6JROE4IQSE2B6IZAVN3D6J3IW4HAHWOSUFTBBQ3QB6XJUBQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.785327870 (2h6m6.566261232s) [noise=true , address=45.91.116.10:4083 , piece_id=4F77NBEDGF4CH45OI3DU5F32NNFNJFJFIJTG6MSM7PMOMASCID2A, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.933440283 (2h6m6.714373652s) [noise=true , address=45.91.116.14:4190 , piece_id=FYVCED2JOP452LAH74LD2G2X4QYK4BQKBQ6E46W5FBWCLIYOXYGQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:52.178035664 (2h6m7.958969033s) [noise=true , address=77.90.31.15:4698 , piece_id=T4YSEM442SNZIQOUPQ2A4XMAJ5AQK6O3A5ID5UNDI6ZENRRDPF7A, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:52.266254515 (2h6m8.047187882s) [noise=true , address=77.90.31.15:4699 , piece_id=UWIGVBJFTBTX7LOAUMZFTVG7ZNNIWC5X62BNRKTUY6337QZF4YGA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:58:08.973792785 (2h6m24.754726157s) [noise=true , address=77.90.31.18:4805 , piece_id=OU5XZPBHVNCWSY3FEH5OAYETFDPZQADQGD6OJC2G7QZZEJJEDW4Q, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:58:20.641348043 (2h6m36.422281419s) [noise=true , address=104.36.82.13:4187 , piece_id=SMJMVDB7S47W3IDN2SU7CI3OLTNUZAGKRSVNPVFBIBZHZHWEPJIQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641410814 (2h6m36.42234418s) [noise=true , address=104.36.82.12:4150 , piece_id=J3XITNETGNAB7LPL5PPNY2VFJPWAQDMTDNN4DWXPZSN2LR6IJ3LA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641457323 (2h6m36.4223907s) [noise=true , address=104.36.82.4:4802 , piece_id=FJBX5ZZNJDPTWKXS6TJDEDFEEIL7HJNRDXHRREKWGWASDYAHAOGQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641500686 (2h6m36.422434061s) [noise=true , address=104.36.82.2:4732 , piece_id=JIPLHZ2S6XEHNR35KRHAPTX6DNLK7TFWKI4X5L5FNQUELJY3N3FQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641546021 (2h6m36.422479388s) [noise=true , address=104.36.82.11:4113 , piece_id=LW2LDEUEPF4RM6JQZMJBXAATTUNQQPN42HH4YDJNR3AE5BYZFE4Q, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641593381 (2h6m36.42252675s) [noise=true , address=104.36.82.8:4969 , piece_id=62UXYO4GV3TZFAFNR6XGFQJJ67PI4MRS76GQNHQRZVUCE4UVPFRA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641666399 (2h6m36.422599768s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:20.641682855 (2h6m36.422616295s) [seg_pos={0 0}, upload=1933]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:58:20.669064353 (2h6m36.449997721s) [upload=1933]: Done uploading segments... Jul 26 23:58:20.669089437 (2h6m36.450022813s) [upload=1933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.029930681s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X...X..X.......X...X....XX..............X.X........X...XX..] duration: 49.623759715s FAIL (2025-07-26 23:58:17 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.623795754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXX] duration: 30.035568219s FAIL (2025-07-26 23:57:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035585152s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [X.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.045183397s FAIL (2025-07-26 23:57:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6OQJ56GL4Z7K7QIN3ZBWX4J4RPG4NDHDLLKUOYRSVY5HISQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.851564451 (2h5m7.632497837s) [noise=true , address=77.90.31.16:4748 , piece_id=QY5S6GF3SMGU7OIZPLSITEH7IV66T2TXAK7VNTNBZRZWWJYAONCA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.851690596 (2h5m7.632623972s) [noise=true , address=104.36.82.47:4521 , piece_id=R25H2RLSDKWE3NJ5YO5WORMUJIEH4CMPQUSCQNWQ57J7CLVUVBVA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852021546 (2h5m7.632954924s) [noise=true , address=64.49.13.138:4482 , piece_id=EXHEP7BWNESA4TPIKVVB7AGSNBSQGCXZZ2HNLLBZGU5WDKLG53QA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852268583 (2h5m7.63320197s) [noise=true , address=77.90.31.18:4821 , piece_id=ANASWX6HVEX4JHDGTLGJJKHQZDSJIFVWJJ4CABNLOFS3FO4BIGMA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852652801 (2h5m7.633586185s) [noise=true , address=104.36.82.55:4199 , piece_id=3FJAV34EDOWTGERP2Z72G3SWIIGW3NSEQKFJ5COTMCEYMM7T423Q, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852709896 (2h5m7.633643281s) [noise=true , address=104.36.82.54:4160 , piece_id=LDGIIUJJX3W52OASO5LTVAVCYLV2RBXJREO2YRVNRHZV2TVMBJPA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853158467 (2h5m7.634091846s) [noise=true , address=45.91.116.9:4277 , piece_id=IUDVNG2CG6G6QTAUSBRR6RLD5FWTLIJ6NL53DNE56BTLCXMXGR7Q, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853180275 (2h5m7.634113662s) [noise=true , address=104.36.82.46:4505 , piece_id=EFMU4SFBKLFSZVS47FXCVMOX7CBMZCW45UQJSKDEV6RSLVZEP3DA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853635476 (2h5m7.634568858s) [noise=true , address=79.127.205.235:4689 , piece_id=XXGET5NG7EWR3Y27Q6ME64ZLYMQ4YXXB56HVRH4JOPG2WV6LZOKA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853752338 (2h5m7.634685713s) [noise=true , address=64.49.13.137:4502 , piece_id=HAH2WCABFGOLFPY3OPYVILV6KDWRUPLS52HDTPLXSLGUTNO72SRA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854060924 (2h5m7.634994312s) [noise=true , address=79.127.201.215:4351 , piece_id=T577FJOHUM53ZGPMZWBFAWU7JJSVZLPMWQPU5KJCIC7EVSXHHFZQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854198788 (2h5m7.635132163s) [noise=true , address=79.127.205.233:4646 , piece_id=FSRHXJS2EKIV6FUOOGB6Y5NIP4PX4Z7RYKQ3PF32P7FBU52ZRDAA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854528885 (2h5m7.635462267s) [noise=true , address=64.49.13.140:4508 , piece_id=6WDQZVIGND7EZ7M7EV4WSBARSROEOUSGQXOVSQZTURJQMENZKPBQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854974085 (2h5m7.635907472s) [noise=true , address=104.36.82.14:4226 , piece_id=ZKQ3M5Y27UPPL6TKG5TJM76VS44AEHCN5UMFQZCDMDU4R37G35NQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.855483525 (2h5m7.636416913s) [noise=true , address=45.91.116.14:4206 , piece_id=G4EX4E4YVNEE4TF5XRALG4MCLQKRWIMTBMOKDQT5TK5HN74CPM6A, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.920993713 (2h5m7.701927097s) [noise=true , address=45.91.116.14:4206 , piece_id=G4EX4E4YVNEE4TF5XRALG4MCLQKRWIMTBMOKDQT5TK5HN74CPM6A, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:51.930652173 (2h5m7.711585536s) [upload=1920]: Next returned nil segment Jul 26 23:56:51.930673404 (2h5m7.711606789s) [upload=1920]: Waiting for error group managing segments... Jul 26 23:56:51.935563179 (2h5m7.716496555s) [noise=true , address=77.90.31.16:4748 , piece_id=QY5S6GF3SMGU7OIZPLSITEH7IV66T2TXAK7VNTNBZRZWWJYAONCA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.455117803 (2h5m8.236051176s) [noise=true , address=79.127.205.233:4646 , piece_id=FSRHXJS2EKIV6FUOOGB6Y5NIP4PX4Z7RYKQ3PF32P7FBU52ZRDAA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.550219414 (2h5m8.331152789s) [noise=true , address=64.49.13.137:4502 , piece_id=HAH2WCABFGOLFPY3OPYVILV6KDWRUPLS52HDTPLXSLGUTNO72SRA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:52.856645765 (2h5m8.637579131s) [noise=true , address=79.127.205.235:4689 , piece_id=XXGET5NG7EWR3Y27Q6ME64ZLYMQ4YXXB56HVRH4JOPG2WV6LZOKA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.864031985 (2h5m8.644965364s) [noise=true , address=79.127.201.215:4351 , piece_id=T577FJOHUM53ZGPMZWBFAWU7JJSVZLPMWQPU5KJCIC7EVSXHHFZQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:53.066732432 (2h5m8.847665809s) [noise=true , address=64.49.13.138:4482 , piece_id=EXHEP7BWNESA4TPIKVVB7AGSNBSQGCXZZ2HNLLBZGU5WDKLG53QA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:53.252768855 (2h5m9.033702234s) [noise=true , address=64.49.13.140:4508 , piece_id=6WDQZVIGND7EZ7M7EV4WSBARSROEOUSGQXOVSQZTURJQMENZKPBQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:05.777632884 (2h5m21.5585664s) [noise=true , address=45.91.116.9:4277 , piece_id=IUDVNG2CG6G6QTAUSBRR6RLD5FWTLIJ6NL53DNE56BTLCXMXGR7Q, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:13.378399729 (2h5m29.159333092s) [noise=true , address=77.90.31.19:4875 , piece_id=YUWMH6OQJ56GL4Z7K7QIN3ZBWX4J4RPG4NDHDLLKUOYRSVY5HISQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:51.782002881 (2h6m7.562936283s) [noise=true , address=77.90.31.18:4821 , piece_id=ANASWX6HVEX4JHDGTLGJJKHQZDSJIFVWJJ4CABNLOFS3FO4BIGMA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:38968->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782091723 (2h6m7.563025098s) [noise=true , address=104.36.82.47:4521 , piece_id=R25H2RLSDKWE3NJ5YO5WORMUJIEH4CMPQUSCQNWQ57J7CLVUVBVA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 159.223.143.39:34394->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782153762 (2h6m7.563087138s) [noise=true , address=104.36.82.54:4160 , piece_id=LDGIIUJJX3W52OASO5LTVAVCYLV2RBXJREO2YRVNRHZV2TVMBJPA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: write tcp 159.223.143.39:43814->104.36.82.54:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782214832 (2h6m7.563148212s) [noise=true , address=104.36.82.14:4226 , piece_id=ZKQ3M5Y27UPPL6TKG5TJM76VS44AEHCN5UMFQZCDMDU4R37G35NQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: write tcp 159.223.143.39:51194->104.36.82.14:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782275917 (2h6m7.563209293s) [noise=true , address=104.36.82.55:4199 , piece_id=3FJAV34EDOWTGERP2Z72G3SWIIGW3NSEQKFJ5COTMCEYMM7T423Q, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 159.223.143.39:44846->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782325578 (2h6m7.563258954s) [noise=true , address=77.90.31.17:4795 , piece_id=4WGI3K74NQS5ZCXCO3UFOI3FQBGHSXIXLGPRLHPLFZT7GMGMZ7HQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782366822 (2h6m7.563300204s) [noise=true , address=104.36.82.46:4505 , piece_id=EFMU4SFBKLFSZVS47FXCVMOX7CBMZCW45UQJSKDEV6RSLVZEP3DA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: rpc: dial tcp 104.36.82.46:4505: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782421930 (2h6m7.56335531s) [noise=true , address=104.36.82.56:4254 , piece_id=N3IX3634VTYN3D5YLTRT2SLDKN2SQBWO567RDEKAHMLMEGKPYFSQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1k4x3HL3PNHwKJX83RyqvnsATHk3HmeMiC12A7raLN58wzvkUN, address:104.36.82.56:4254): protocol: rpc: dial tcp 104.36.82.56:4254: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782483496 (2h6m7.563416938s) [seg_pos={0 0}, upload=1920]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:57:51.824438035 (2h6m7.605371414s) [upload=1920]: Done uploading segments... Jul 26 23:57:51.824462856 (2h6m7.605396225s) [upload=1920]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.045205187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 30.027203231s FAIL (2025-07-26 23:57:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420354433 (2h5m36.201287809s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420374692 (2h5m36.201308072s) [seg_pos={0 0}, upload=1919]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:57:20.455540937 (2h5m36.236474325s) [upload=1919]: Done uploading segments... Jul 26 23:57:20.455569078 (2h5m36.236502457s) [upload=1919]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:57:20.568454898 (2h5m36.349388274s) [upload=1927]: Starting upload Jul 26 23:57:20.568669185 (2h5m36.349602567s) [upload=1927]: Done starting upload Jul 26 23:57:20.568792233 (2h5m36.349725603s) [upload=1927]: Uploading segments... Jul 26 23:57:20.569146906 (2h5m36.35008029s) [upload=1927]: Got next segment. Inline: false Jul 26 23:57:20.569244299 (2h5m36.350177684s) [upload=1927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:20.569251429 (2h5m36.350184819s) [upload=1927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:20.601609712 (2h5m36.382543099s) [seg_pos={0 0}, upload=1927]: Begin upload segment... Jul 26 23:57:20.601673862 (2h5m36.382607237s) [seg_pos={0 0}, upload=1927]: Done begin upload segment. Jul 26 23:57:20.601715768 (2h5m36.382649134s) [noise=true , address=104.36.82.6:4876 , piece_id=637YSFHNBIV4P5XWRCPH24LPJYY3M6MFFWC4Y5X6Z655ZZPGXJ6Q, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.602124165 (2h5m36.383057548s) [noise=true , address=45.91.116.14:4189 , piece_id=KLE43JQOKTKITXNAJJVJYFMFCJ3SFSMC3EMLBFXK6HC3TCF5MJZA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.602487258 (2h5m36.383420644s) [noise=true , address=45.91.116.13:4153 , piece_id=3GBGT5TMQO3ARTKZVA3SJQ6LNOVZTVUEASYVENEJAQGJEGPRSOCA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603068947 (2h5m36.384002327s) [noise=true , address=77.90.31.19:4842 , piece_id=PSQZGLVQKUU3DMIOT3VQ62XZAFVLSGFQ66Z7PMBASX6KZPJ522IQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603200750 (2h5m36.384134127s) [noise=true , address=45.91.116.4:4938 , piece_id=JH3GTJD37UAFNHJEE7POZI2BYESIUV3SLVHEVTYLWJUECKT3MBVA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603794398 (2h5m36.384727774s) [noise=true , address=77.90.31.14:4662 , piece_id=VWABP3RH3VT2SQA3POY5WTC2TDRR7PXSRSWIYOXKEUG2WRIDBIJA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.604033107 (2h5m36.384966494s) [noise=true , address=104.36.82.5:4837 , piece_id=UH67FWYQTAYDGTP65NA5BO2MSQIOSPGSJYYR6GYP55C7JR5EIGGQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.604891309 (2h5m36.385824686s) [noise=true , address=45.91.116.8:4048 , piece_id=P2UZRXYEXUTUTQMUEUOYKOXCWDUGQXWDPRJ2PXX35DMIYXANKGJQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.605266931 (2h5m36.386200313s) [noise=true , address=77.90.31.13:4626 , piece_id=75CQFMO5CTU7RZDFQJ5U3FN3GGSTQAHYYF4EE4YDRBPBLGRJDVRA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.605643773 (2h5m36.386577221s) [noise=true , address=77.90.31.15:4699 , piece_id=7CZYIG3GWGSDKLXUOG35Y5PHK46MKPUWWWCEH4XOBCRKN2MBW7JA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.606028106 (2h5m36.386961491s) [noise=true , address=45.91.116.11:4117 , piece_id=6MS4BJB4KL3RQL5AOKLZHZ4T5BJR2M4LK4ZZ6H5R4OOAHZ7TSGSA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.606380314 (2h5m36.387313697s) [noise=true , address=45.91.116.12:4225 , piece_id=ORCWODLYVGYKU6YMJH2W2MPN4DMNFJFB22VY6ZWER7WLXBBVTGLQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607164117 (2h5m36.388097501s) [noise=true , address=77.90.31.17:4771 , piece_id=VVEBB6CJDW3CYLC6HM2MGCAGDXM7T5ZOHOMQXA6V3WBUI34Q3RQQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607560464 (2h5m36.388493854s) [noise=true , address=104.36.82.7:4912 , piece_id=GFNR5C5LGCJEOOWZ52RNSU2HHJPN5I3SRAAOIX3WSCYVGHIR5QQQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607983625 (2h5m36.388917012s) [noise=true , address=77.90.31.12:4589 , piece_id=P3NL3JACSRBQ6BQZDMENVSFLHWEPVI47HRCFHUYI25QXLW2XBTRQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.608581577 (2h5m36.389514953s) [noise=true , address=45.91.116.5:4299 , piece_id=IT3CWTXUVLJCZHH3SE6R6S2SESH6WCPIK436YS26DWPGNNEA2MXQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.703573074 (2h5m36.484506464s) [upload=1927]: Next returned nil segment Jul 26 23:57:20.703595449 (2h5m36.484528836s) [upload=1927]: Waiting for error group managing segments... Jul 26 23:57:20.707832208 (2h5m36.488765575s) [noise=true , address=45.91.116.5:4299 , piece_id=IT3CWTXUVLJCZHH3SE6R6S2SESH6WCPIK436YS26DWPGNNEA2MXQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.709464304 (2h5m36.490397669s) [noise=true , address=77.90.31.14:4662 , piece_id=VWABP3RH3VT2SQA3POY5WTC2TDRR7PXSRSWIYOXKEUG2WRIDBIJA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.709931221 (2h5m36.490864584s) [noise=true , address=45.91.116.12:4225 , piece_id=ORCWODLYVGYKU6YMJH2W2MPN4DMNFJFB22VY6ZWER7WLXBBVTGLQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.710397031 (2h5m36.491330401s) [noise=true , address=45.91.116.4:4938 , piece_id=JH3GTJD37UAFNHJEE7POZI2BYESIUV3SLVHEVTYLWJUECKT3MBVA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.710932201 (2h5m36.491865571s) [noise=true , address=45.91.116.11:4117 , piece_id=6MS4BJB4KL3RQL5AOKLZHZ4T5BJR2M4LK4ZZ6H5R4OOAHZ7TSGSA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.711462379 (2h5m36.492395744s) [noise=true , address=45.91.116.8:4048 , piece_id=P2UZRXYEXUTUTQMUEUOYKOXCWDUGQXWDPRJ2PXX35DMIYXANKGJQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.711871688 (2h5m36.492805057s) [noise=true , address=45.91.116.13:4153 , piece_id=3GBGT5TMQO3ARTKZVA3SJQ6LNOVZTVUEASYVENEJAQGJEGPRSOCA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.712409304 (2h5m36.493342674s) [noise=true , address=77.90.31.15:4699 , piece_id=7CZYIG3GWGSDKLXUOG35Y5PHK46MKPUWWWCEH4XOBCRKN2MBW7JA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.757155063 (2h5m36.538088444s) [noise=true , address=45.91.116.14:4189 , piece_id=KLE43JQOKTKITXNAJJVJYFMFCJ3SFSMC3EMLBFXK6HC3TCF5MJZA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.061385227 (2h5m37.842318606s) [noise=true , address=77.90.31.12:4589 , piece_id=P3NL3JACSRBQ6BQZDMENVSFLHWEPVI47HRCFHUYI25QXLW2XBTRQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.141938960 (2h5m37.922872339s) [noise=true , address=77.90.31.13:4626 , piece_id=75CQFMO5CTU7RZDFQJ5U3FN3GGSTQAHYYF4EE4YDRBPBLGRJDVRA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.740247145 (2h5m38.521180511s) [noise=true , address=77.90.31.19:4842 , piece_id=PSQZGLVQKUU3DMIOT3VQ62XZAFVLSGFQ66Z7PMBASX6KZPJ522IQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:50.538784063 (2h6m6.319717437s) [noise=true , address=77.90.31.17:4771 , piece_id=VVEBB6CJDW3CYLC6HM2MGCAGDXM7T5ZOHOMQXA6V3WBUI34Q3RQQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:44402->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538872737 (2h6m6.319806102s) [noise=true , address=104.36.82.6:4876 , piece_id=637YSFHNBIV4P5XWRCPH24LPJYY3M6MFFWC4Y5X6Z655ZZPGXJ6Q, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538916585 (2h6m6.319849954s) [noise=true , address=104.36.82.7:4912 , piece_id=GFNR5C5LGCJEOOWZ52RNSU2HHJPN5I3SRAAOIX3WSCYVGHIR5QQQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538961473 (2h6m6.319894837s) [noise=true , address=104.36.82.5:4837 , piece_id=UH67FWYQTAYDGTP65NA5BO2MSQIOSPGSJYYR6GYP55C7JR5EIGGQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:36770->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538990054 (2h6m6.319923431s) [seg_pos={0 0}, upload=1927]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:57:50.564064323 (2h6m6.344997692s) [upload=1927]: Done uploading segments... Jul 26 23:57:50.564195664 (2h6m6.345129035s) [upload=1927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027222085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.X.X.......XXX...XX.....X.XXX...............X..X......X.] duration: 538.491978ms FAIL (2025-07-26 23:57:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.525264ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXX] duration: 30.030452728s FAIL (2025-07-26 23:57:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030499873s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.X.X.X.......XXX...XX.....X.XXX...............X..X......X] duration: 30.00219189s FAIL (2025-07-26 23:57:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00222092s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 30.037657313s FAIL (2025-07-26 23:57:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== al tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902366849 (2h4m24.683300229s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902415935 (2h4m24.68334932s) [seg_pos={0 0}, upload=1901]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:56:08.929119953 (2h4m24.710053318s) [upload=1901]: Done uploading segments... Jul 26 23:56:08.929245899 (2h4m24.710179269s) [upload=1901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:56:50.465037803 (2h5m6.245971187s) [upload=1919]: Starting upload Jul 26 23:56:50.465161980 (2h5m6.246095359s) [upload=1919]: Done starting upload Jul 26 23:56:50.465414494 (2h5m6.24634787s) [upload=1919]: Uploading segments... Jul 26 23:56:50.465592479 (2h5m6.246525864s) [upload=1919]: Got next segment. Inline: false Jul 26 23:56:50.465700612 (2h5m6.246633992s) [upload=1919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:56:50.465707959 (2h5m6.246641338s) [upload=1919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:56:50.498627491 (2h5m6.279560883s) [seg_pos={0 0}, upload=1919]: Begin upload segment... Jul 26 23:56:50.498707953 (2h5m6.27964135s) [seg_pos={0 0}, upload=1919]: Done begin upload segment. Jul 26 23:56:50.498742575 (2h5m6.279675962s) [noise=true , address=45.91.116.10:4082 , piece_id=NPWZ36AX4YZXZFWHIGDW63MY2MRF4UOAMZMPW5JWITMLMPA6LLFA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.499388568 (2h5m6.280321964s) [noise=true , address=77.90.31.21:4879 , piece_id=7APMPPEDNMZZLE3JFPCGDWRLTU7JZCKK7RI77HXHEZZ7JVABZZIQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.499992313 (2h5m6.280925701s) [noise=true , address=104.36.82.5:4839 , piece_id=TNVZP4Z4MCOP4MPKVXCUYDQFYJKYLYTJQ35HU4Y37FYFM6UQOWDQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.500649200 (2h5m6.281582589s) [noise=true , address=45.91.116.7:4009 , piece_id=7DGFXBBQ6NDDX2VXIDC3CZIC62QK6GSXODJ6SWQL7FTVCWSJD3CA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.501161510 (2h5m6.282094893s) [noise=true , address=77.90.31.19:4842 , piece_id=XERXEWTVI7AZ7CFETWVSSCDG7AAOK72DE7YNV6LR2EJNSANATZIA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.501647599 (2h5m6.282580985s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502106967 (2h5m6.283040356s) [noise=true , address=45.91.116.8:4047 , piece_id=YGJBQCFZMDVUWR7AXFOYVQWY7E3JKOTBBC3VFK6YIYL6I2QZMALQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502504476 (2h5m6.283437864s) [noise=true , address=45.91.116.6:4975 , piece_id=LEB4XACHU3ZUCJ22DYAUBJ4UUQJUKXJTPBGSVE54A5KVVNXUSWLA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502912525 (2h5m6.283845937s) [noise=true , address=104.36.82.4:4804 , piece_id=EFQ2RTSQRSU5ZRTRAFMJC3YHX4NG4EQHAHKDMME3JA4ZJI5PCJNA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.503298528 (2h5m6.284231926s) [noise=true , address=45.91.116.7:4010 , piece_id=5TPYUI2N5FK5MWIVJKDDIYU2YHG2LQIXM7LHGRKS4GA4QVSWBQIQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.503685240 (2h5m6.284618628s) [noise=true , address=45.91.116.4:4939 , piece_id=XJZOTMKIXTY5TTDHHJZHBYZHJVXDPMKWR7UOMLLCC7ABFTYMVPGQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504060007 (2h5m6.284993397s) [noise=true , address=104.36.82.16:4294 , piece_id=GMBE2WSJIBVA2KNPVQGXIZY5ING5NMRBQLNAIJK3L5AEAHYK4DLA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504460154 (2h5m6.285393542s) [noise=true , address=45.91.116.14:4189 , piece_id=ZRGTEUP4WOTY56VPDWI5TF4W2FT6KGLW67SID7KE2DZ57EBM4F2Q, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504849562 (2h5m6.28578295s) [noise=true , address=77.90.31.12:4592 , piece_id=P2J5ODVVKQ4KM52YE4OSGF2THNSTTCY7TZTW6RGS3BO2Q6C4FLXA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.505234196 (2h5m6.286167585s) [noise=true , address=77.90.31.14:4661 , piece_id=WHMJRFJQZMKNB7FHO3JPAVJ7IOHK4XVXX6JAZUQYJQVR6SXN6N6A, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.505589115 (2h5m6.286522504s) [noise=true , address=45.91.116.12:4225 , piece_id=U3W36JTBMJVYWOQRQHMAHGJ5IBY73N44B5W5YK7B3OC5AE65R5PQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.611080234 (2h5m6.392013621s) [upload=1919]: Next returned nil segment Jul 26 23:56:50.611099550 (2h5m6.392032938s) [upload=1919]: Waiting for error group managing segments... Jul 26 23:56:50.614140486 (2h5m6.395073864s) [noise=true , address=45.91.116.4:4939 , piece_id=XJZOTMKIXTY5TTDHHJZHBYZHJVXDPMKWR7UOMLLCC7ABFTYMVPGQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.614630565 (2h5m6.395563942s) [noise=true , address=45.91.116.7:4010 , piece_id=5TPYUI2N5FK5MWIVJKDDIYU2YHG2LQIXM7LHGRKS4GA4QVSWBQIQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.615011656 (2h5m6.395945024s) [noise=true , address=45.91.116.10:4082 , piece_id=NPWZ36AX4YZXZFWHIGDW63MY2MRF4UOAMZMPW5JWITMLMPA6LLFA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.616327443 (2h5m6.397260815s) [noise=true , address=45.91.116.7:4009 , piece_id=7DGFXBBQ6NDDX2VXIDC3CZIC62QK6GSXODJ6SWQL7FTVCWSJD3CA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.616952065 (2h5m6.397885431s) [noise=true , address=77.90.31.14:4661 , piece_id=WHMJRFJQZMKNB7FHO3JPAVJ7IOHK4XVXX6JAZUQYJQVR6SXN6N6A, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.617571736 (2h5m6.398505113s) [noise=true , address=45.91.116.8:4047 , piece_id=YGJBQCFZMDVUWR7AXFOYVQWY7E3JKOTBBC3VFK6YIYL6I2QZMALQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.618080351 (2h5m6.399013728s) [noise=true , address=45.91.116.6:4975 , piece_id=LEB4XACHU3ZUCJ22DYAUBJ4UUQJUKXJTPBGSVE54A5KVVNXUSWLA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.618708942 (2h5m6.399642305s) [noise=true , address=45.91.116.12:4225 , piece_id=U3W36JTBMJVYWOQRQHMAHGJ5IBY73N44B5W5YK7B3OC5AE65R5PQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.782869602 (2h5m6.563802983s) [noise=true , address=45.91.116.14:4189 , piece_id=ZRGTEUP4WOTY56VPDWI5TF4W2FT6KGLW67SID7KE2DZ57EBM4F2Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:52.212986258 (2h5m7.993919628s) [noise=true , address=77.90.31.12:4592 , piece_id=P2J5ODVVKQ4KM52YE4OSGF2THNSTTCY7TZTW6RGS3BO2Q6C4FLXA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:52.748770297 (2h5m8.529703659s) [noise=true , address=77.90.31.19:4842 , piece_id=XERXEWTVI7AZ7CFETWVSSCDG7AAOK72DE7YNV6LR2EJNSANATZIA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:57:20.419852898 (2h5m36.200786278s) [noise=true , address=104.36.82.4:4804 , piece_id=EFQ2RTSQRSU5ZRTRAFMJC3YHX4NG4EQHAHKDMME3JA4ZJI5PCJNA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420263082 (2h5m36.20119646s) [noise=true , address=77.90.31.21:4879 , piece_id=7APMPPEDNMZZLE3JFPCGDWRLTU7JZCKK7RI77HXHEZZ7JVABZZIQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420279492 (2h5m36.201212856s) [noise=true , address=104.36.82.5:4839 , piece_id=TNVZP4Z4MCOP4MPKVXCUYDQFYJKYLYTJQ35HU4Y37FYFM6UQOWDQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420343148 (2h5m36.201276527s) [noise=true , address=104.36.82.16:4294 , piece_id=GMBE2WSJIBVA2KNPVQGXIZY5ING5NMRBQLNAIJK3L5AEAHYK4DLA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420354433 (2h5m36.201287809s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420374692 (2h5m36.201308072s) [seg_pos={0 0}, upload=1919]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:57:20.455540937 (2h5m36.236474325s) [upload=1919]: Done uploading segments... Jul 26 23:57:20.455569078 (2h5m36.236502457s) [upload=1919]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037752922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.024663066s FAIL (2025-07-26 23:56:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== one begin upload segment. Jul 26 23:55:51.736607196 (2h4m7.51754058s) [noise=true , address=64.49.13.138:4482 , piece_id=QAMY2AP34W3YL4EWMFBI4PP5EB2QMCCMJ5RUBK6Y4YZKSKPDHH6Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737016986 (2h4m7.517950368s) [noise=true , address=104.36.82.47:4531 , piece_id=4V4W23MOJSCA5MVBLK3S47KTL4TCCISLQPSELV73THB7NQU25VHA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737265194 (2h4m7.51819857s) [noise=true , address=79.127.205.233:4640 , piece_id=PV6A4TGBW3XUVVM57YLHCD27KXZFNY2NMSJKSHMXVMLMI73FYPUQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737484924 (2h4m7.518418299s) [noise=true , address=104.36.82.54:4191 , piece_id=H4CQZ6G25HC62K73BF7PVYPNYLI34HVHOPZ4HXAP4TM6Q6GXQMCA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737709062 (2h4m7.51864244s) [noise=true , address=77.90.31.17:4784 , piece_id=QBKYDSDH4OOR2GGNBGNPNY5JQND5LRTJPKYID22KUW6CRSU6KWEQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737932745 (2h4m7.518866126s) [noise=true , address=79.127.205.234:4614 , piece_id=QIXRNCPDPIV7K7SJS4XC3RPGTLYBM34EDL7OTX6TJVUCI6LSOI6Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738153974 (2h4m7.519087352s) [noise=true , address=79.127.219.38:4018 , piece_id=NSB4GJIUQ6XF7VPSP4RLR4YADNIXAVWIMFDVUKSSXVZFTH6G25BQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738372573 (2h4m7.519305954s) [noise=true , address=79.127.201.215:4359 , piece_id=4CAYQOW5UOSPDAQUGD7EXKNEJZLVSI5GDUWL5TW4AUGJ2WDRNNYQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738591768 (2h4m7.519525145s) [noise=true , address=45.91.116.9:4277 , piece_id=D6RHI7YTRLHQU7C3VYKQJSHPZLNPIHI7C7K6UHG6HRQVZFTK5ORQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738826597 (2h4m7.519759978s) [noise=true , address=104.36.82.46:4484 , piece_id=OQTCIP4MUT4PAK2VT5AS4DMYDXDIK3HUKFL3CCEPS3NCCD44LM4Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739097434 (2h4m7.520030815s) [noise=true , address=77.90.31.10:4531 , piece_id=ZRDLEG5CWJCEXFZ346CQP6ZMSFBIVW4ZHZDTCR7TJZ6BOELZP5XQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739314856 (2h4m7.520248217s) [noise=true , address=79.127.220.98:4422 , piece_id=U4IQG7XK2WJ6YZRUKZYT5CRPO63QABH4WWVGUSJCMNCSCJHQANRA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739532946 (2h4m7.520466327s) [noise=true , address=104.36.82.52:4109 , piece_id=KZR6YMYHWEZSFLIA5SETSAPGYLRRTDB5IW6NLZRJN2NKO66F7WFA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739757421 (2h4m7.520690804s) [noise=true , address=104.36.82.53:4141 , piece_id=CVLR3VT4YOMO2E4FI42GQUAEPMZY5LT2OVOXCPMLIJM6U6PV7HIQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739993694 (2h4m7.520927086s) [noise=true , address=77.90.31.13:4631 , piece_id=I7ZA4OA4UEGRPCZ5UKF2TPXC6YUADPZXR4YQPJBNDTDDAFYUPTUQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740388551 (2h4m7.521321929s) [noise=true , address=64.49.13.138:4483 , piece_id=NM3MXV7YQ223YSH6SRD6UGMLFYSCPBOP7LC7PRTDRJYCHMPX4SWQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740696228 (2h4m7.521629609s) [noise=true , address=77.90.31.21:4887 , piece_id=TVZAO6KLHQGPTNV3OGREIEPB3CIWXR4L6ZYTP443BSF47TM5Z5JA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740983229 (2h4m7.521916608s) [noise=true , address=77.90.31.19:4871 , piece_id=4IFKVUY5NSUOQCGA54MFILJRBYTP7U47CMQJ56LMCLOYIOJ5PQFA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.809810613 (2h4m7.590743981s) [upload=1906]: Next returned nil segment Jul 26 23:55:51.809828216 (2h4m7.590761582s) [upload=1906]: Waiting for error group managing segments... Jul 26 23:55:51.813341620 (2h4m7.594274983s) [noise=true , address=79.127.219.38:4018 , piece_id=NSB4GJIUQ6XF7VPSP4RLR4YADNIXAVWIMFDVUKSSXVZFTH6G25BQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:52.334918607 (2h4m8.115851975s) [noise=true , address=79.127.205.234:4614 , piece_id=QIXRNCPDPIV7K7SJS4XC3RPGTLYBM34EDL7OTX6TJVUCI6LSOI6Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:52.701829381 (2h4m8.482762755s) [noise=true , address=79.127.205.233:4640 , piece_id=PV6A4TGBW3XUVVM57YLHCD27KXZFNY2NMSJKSHMXVMLMI73FYPUQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:53.063013429 (2h4m8.843946794s) [noise=true , address=77.90.31.13:4631 , piece_id=I7ZA4OA4UEGRPCZ5UKF2TPXC6YUADPZXR4YQPJBNDTDDAFYUPTUQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.328943719 (2h4m10.10987708s) [noise=true , address=79.127.201.215:4359 , piece_id=4CAYQOW5UOSPDAQUGD7EXKNEJZLVSI5GDUWL5TW4AUGJ2WDRNNYQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.663246223 (2h4m10.444179592s) [noise=true , address=77.90.31.19:4871 , piece_id=4IFKVUY5NSUOQCGA54MFILJRBYTP7U47CMQJ56LMCLOYIOJ5PQFA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.761753694 (2h4m10.54268706s) [noise=true , address=64.49.13.138:4483 , piece_id=NM3MXV7YQ223YSH6SRD6UGMLFYSCPBOP7LC7PRTDRJYCHMPX4SWQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:55:56.372494244 (2h4m12.15342761s) [noise=true , address=79.127.220.98:4422 , piece_id=U4IQG7XK2WJ6YZRUKZYT5CRPO63QABH4WWVGUSJCMNCSCJHQANRA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:59.102101660 (2h4m14.883035046s) [noise=true , address=64.49.13.138:4482 , piece_id=QAMY2AP34W3YL4EWMFBI4PP5EB2QMCCMJ5RUBK6Y4YZKSKPDHH6Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:56:00.743187561 (2h4m16.52412093s) [noise=true , address=45.91.116.9:4277 , piece_id=D6RHI7YTRLHQU7C3VYKQJSHPZLNPIHI7C7K6UHG6HRQVZFTK5ORQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:56:51.669324069 (2h5m7.450257455s) [noise=true , address=77.90.31.10:4531 , piece_id=ZRDLEG5CWJCEXFZ346CQP6ZMSFBIVW4ZHZDTCR7TJZ6BOELZP5XQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:56:51.669559077 (2h5m7.450492452s) [noise=true , address=104.36.82.53:4141 , piece_id=CVLR3VT4YOMO2E4FI42GQUAEPMZY5LT2OVOXCPMLIJM6U6PV7HIQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669616415 (2h5m7.450549779s) [noise=true , address=77.90.31.21:4887 , piece_id=TVZAO6KLHQGPTNV3OGREIEPB3CIWXR4L6ZYTP443BSF47TM5Z5JA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669665917 (2h5m7.450599293s) [noise=true , address=104.36.82.54:4191 , piece_id=H4CQZ6G25HC62K73BF7PVYPNYLI34HVHOPZ4HXAP4TM6Q6GXQMCA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669709557 (2h5m7.450642923s) [noise=true , address=104.36.82.46:4484 , piece_id=OQTCIP4MUT4PAK2VT5AS4DMYDXDIK3HUKFL3CCEPS3NCCD44LM4Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669717210 (2h5m7.450650577s) [noise=true , address=104.36.82.52:4109 , piece_id=KZR6YMYHWEZSFLIA5SETSAPGYLRRTDB5IW6NLZRJN2NKO66F7WFA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 159.223.143.39:45574->104.36.82.52:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669774429 (2h5m7.450707805s) [noise=true , address=77.90.31.17:4784 , piece_id=QBKYDSDH4OOR2GGNBGNPNY5JQND5LRTJPKYID22KUW6CRSU6KWEQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: rpc: dial tcp 77.90.31.17:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669898558 (2h5m7.450831937s) [noise=true , address=104.36.82.47:4531 , piece_id=4V4W23MOJSCA5MVBLK3S47KTL4TCCISLQPSELV73THB7NQU25VHA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: write tcp 159.223.143.39:50378->104.36.82.47:4531: use of closed network connection; context deadline exceeded Jul 26 23:56:51.669969401 (2h5m7.450902785s) [seg_pos={0 0}, upload=1906]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:56:51.692064813 (2h5m7.472998195s) [upload=1906]: Done uploading segments... Jul 26 23:56:51.692195106 (2h5m7.473128488s) [upload=1906]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.024683276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.X...X..X.......X...X....XX..............X.X........X...XX.] duration: 23.929279306s FAIL (2025-07-26 23:56:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.929299387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXX] duration: 30.036715565s FAIL (2025-07-26 23:56:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.036757719s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XX.X...X..X.......X...X....XX..............X.X........X...XX] duration: 1m0.001256143s FAIL (2025-07-26 23:56:20 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001308454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 30.031576958s FAIL (2025-07-26 23:56:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ose: conn closed: op Jul 26 23:55:38.792887535 (2h3m54.57382091s) [seg_pos={0 0}, upload=1891]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:55:38.815750378 (2h3m54.596683745s) [upload=1891]: Done uploading segments... Jul 26 23:55:38.815795701 (2h3m54.596729078s) [upload=1891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:55:38.943289758 (2h3m54.724223134s) [upload=1901]: Starting upload Jul 26 23:55:38.943363577 (2h3m54.724296944s) [upload=1901]: Done starting upload Jul 26 23:55:38.943438500 (2h3m54.724371867s) [upload=1901]: Uploading segments... Jul 26 23:55:38.943711796 (2h3m54.724645161s) [upload=1901]: Got next segment. Inline: false Jul 26 23:55:38.943745315 (2h3m54.724678679s) [upload=1901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:38.943750314 (2h3m54.724683689s) [upload=1901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:38.972810375 (2h3m54.753743783s) [seg_pos={0 0}, upload=1901]: Begin upload segment... Jul 26 23:55:38.972864526 (2h3m54.753797908s) [seg_pos={0 0}, upload=1901]: Done begin upload segment. Jul 26 23:55:38.972896174 (2h3m54.753829562s) [noise=true , address=45.91.116.13:4155 , piece_id=FAHGE7ZDRZGVP2QIXZR346CTB56RSL4V5RX3PUBPYMUQE6HFSWPQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.972924889 (2h3m54.753858283s) [noise=true , address=45.91.116.4:4940 , piece_id=2JM4LX2ZP35OI6TENM27UVJSKLTWAR5F3BN7BECXB4FJJNGCFQAQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973371247 (2h3m54.754304632s) [noise=true , address=45.91.116.6:4975 , piece_id=5ZHK2PU6U3PSDKSN63AFT7IF7QYMLGGULCEM5V3DSOU4E2VCAAHQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973440265 (2h3m54.754373647s) [noise=true , address=104.36.82.7:4909 , piece_id=OZ2OLA43OA6OVWHQFHDROJMCRMESPTQZN5ZBI4YEZ33KIIZYCQDQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973791860 (2h3m54.754725243s) [noise=true , address=45.91.116.9:4263 , piece_id=Z3LZWOD5AG7N3TEZVZFEUYI4DDCIKVBE24ZW3X37VQXYJLBQPTSA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973838543 (2h3m54.754771927s) [noise=true , address=104.36.82.9:4005 , piece_id=M74RQT63CRR4G3FVH45K4OCFAFDV5CGCGYQHPRNLHLXSAUMQ4TMQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974147960 (2h3m54.755081344s) [noise=true , address=104.36.82.2:4731 , piece_id=YBX6D6PUX6ZB5QSQT6FKSTRHT3W62UEUPFUBCJSA5BTF4ERI5DCA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974245384 (2h3m54.755178766s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974462553 (2h3m54.755395938s) [noise=true , address=104.36.82.10:4079 , piece_id=M7KQW7PUVSTGLYDSFSK6R4ZT5M7GKBOBLKJOWX6JVUSQXK4DRWUA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974693632 (2h3m54.755627032s) [noise=true , address=45.91.116.7:4010 , piece_id=T4UPZISO5ZZUXH4NOHNLREZVON74NYFXT6P45BNKNOVGK52T23KQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974763451 (2h3m54.755696827s) [noise=true , address=104.36.82.2:4730 , piece_id=BCS7UQ27PBQLBAJ6F6PTLRES5C3B4WHW73YIOERGLWUUAJIZRGTQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975102163 (2h3m54.756035544s) [noise=true , address=104.36.82.11:4114 , piece_id=LUKGOXOYPYHPBK63A2QMJUECNURLPE5THPH4DWMETCCDVTZKCL3A, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975244877 (2h3m54.756178259s) [noise=true , address=45.91.116.5:4298 , piece_id=JECAKQX4EUZ57QWUG5WUNH4DMXGATTS57CSSO5I2BREGTJCK375A, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975405189 (2h3m54.75633857s) [noise=true , address=77.90.31.14:4663 , piece_id=P6SPJFZIIQKBMLPBURURGINL7HMMSG2IJXQ7ETINRVYENPXACRQQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975634674 (2h3m54.75656806s) [noise=true , address=45.91.116.9:4262 , piece_id=DHP5JK2XWZDV7NP45CJYA57E7TFUDTXARNBASWTNGWJVK3OQW7DA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975703630 (2h3m54.756637006s) [noise=true , address=45.91.116.8:4045 , piece_id=V4RXNHVDSMTLIVCZMN4SDW7HLMWCCIOEUZZJ5ZX3J4RSYVUZ6OFQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:39.067399969 (2h3m54.848333337s) [upload=1901]: Next returned nil segment Jul 26 23:55:39.067415508 (2h3m54.848348883s) [upload=1901]: Waiting for error group managing segments... Jul 26 23:55:39.071290081 (2h3m54.852223456s) [noise=true , address=45.91.116.13:4155 , piece_id=FAHGE7ZDRZGVP2QIXZR346CTB56RSL4V5RX3PUBPYMUQE6HFSWPQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.071840948 (2h3m54.852774317s) [noise=true , address=45.91.116.7:4010 , piece_id=T4UPZISO5ZZUXH4NOHNLREZVON74NYFXT6P45BNKNOVGK52T23KQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.072481061 (2h3m54.853414438s) [noise=true , address=45.91.116.4:4940 , piece_id=2JM4LX2ZP35OI6TENM27UVJSKLTWAR5F3BN7BECXB4FJJNGCFQAQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.072992218 (2h3m54.853925594s) [noise=true , address=45.91.116.5:4298 , piece_id=JECAKQX4EUZ57QWUG5WUNH4DMXGATTS57CSSO5I2BREGTJCK375A, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073391224 (2h3m54.854324593s) [noise=true , address=45.91.116.6:4975 , piece_id=5ZHK2PU6U3PSDKSN63AFT7IF7QYMLGGULCEM5V3DSOU4E2VCAAHQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073736742 (2h3m54.854670116s) [noise=true , address=77.90.31.14:4663 , piece_id=P6SPJFZIIQKBMLPBURURGINL7HMMSG2IJXQ7ETINRVYENPXACRQQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073827343 (2h3m54.854760709s) [noise=true , address=45.91.116.8:4045 , piece_id=V4RXNHVDSMTLIVCZMN4SDW7HLMWCCIOEUZZJ5ZX3J4RSYVUZ6OFQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.504917220 (2h3m55.28585059s) [noise=true , address=45.91.116.9:4262 , piece_id=DHP5JK2XWZDV7NP45CJYA57E7TFUDTXARNBASWTNGWJVK3OQW7DA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.506696337 (2h3m55.287629708s) [noise=true , address=45.91.116.9:4263 , piece_id=Z3LZWOD5AG7N3TEZVZFEUYI4DDCIKVBE24ZW3X37VQXYJLBQPTSA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:56:08.902025840 (2h4m24.682959207s) [noise=true , address=104.36.82.11:4114 , piece_id=LUKGOXOYPYHPBK63A2QMJUECNURLPE5THPH4DWMETCCDVTZKCL3A, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902162029 (2h4m24.683095399s) [noise=true , address=104.36.82.2:4731 , piece_id=YBX6D6PUX6ZB5QSQT6FKSTRHT3W62UEUPFUBCJSA5BTF4ERI5DCA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:55760->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902193436 (2h4m24.683126824s) [noise=true , address=104.36.82.2:4730 , piece_id=BCS7UQ27PBQLBAJ6F6PTLRES5C3B4WHW73YIOERGLWUUAJIZRGTQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902268315 (2h4m24.683201687s) [noise=true , address=104.36.82.7:4909 , piece_id=OZ2OLA43OA6OVWHQFHDROJMCRMESPTQZN5ZBI4YEZ33KIIZYCQDQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902280088 (2h4m24.68321345s) [noise=true , address=104.36.82.9:4005 , piece_id=M74RQT63CRR4G3FVH45K4OCFAFDV5CGCGYQHPRNLHLXSAUMQ4TMQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902339682 (2h4m24.683273049s) [noise=true , address=104.36.82.10:4079 , piece_id=M7KQW7PUVSTGLYDSFSK6R4ZT5M7GKBOBLKJOWX6JVUSQXK4DRWUA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902366849 (2h4m24.683300229s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902415935 (2h4m24.68334932s) [seg_pos={0 0}, upload=1901]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:56:08.929119953 (2h4m24.710053318s) [upload=1901]: Done uploading segments... Jul 26 23:56:08.929245899 (2h4m24.710179269s) [upload=1901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.031598806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXX] duration: 30.025334716s FAIL (2025-07-26 23:56:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025352168s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX.X.X.X.......XXX...XX.....X.XXX...............X..X......] duration: 2.625398478s FAIL (2025-07-26 23:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.625448942s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.034451975s FAIL (2025-07-26 23:55:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 04 (2h3m7.392798688s) [seg_pos={0 0}, upload=1886]: Done begin upload segment. Jul 26 23:54:51.611900165 (2h3m7.392833551s) [noise=true , address=77.90.31.17:4780 , piece_id=MXK24WZJGMPX3THAEEJDCVCGBB2PX3TBFSM6MQ5H43VUV3RN243A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.611917856 (2h3m7.392851242s) [noise=true , address=104.36.82.53:4155 , piece_id=XN5QZ2S5UI32H4C6X7E6WOSCDS3WBNZ5DKDTKR2N2O4DHUKH4MPQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612497958 (2h3m7.393431349s) [noise=true , address=79.127.201.214:4321 , piece_id=PEQ4RK3LDREIGED2ZZFUPRE57XH4KD3VHHKAW7C7FHYCZXEQW3TQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612621971 (2h3m7.393555357s) [noise=true , address=79.127.205.245:4249 , piece_id=2GGDQGAWBTBUOQLUPLZU7FKD4WS7AKL6LNNOV5HCHE7RENRURQFA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612948269 (2h3m7.393881652s) [noise=true , address=79.127.205.246:4278 , piece_id=LX2ENB6LMVMG2ZUKG32ZV2WZ5JKZPKHMCK6T62WBDRC73LFV6I6A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613213445 (2h3m7.394146832s) [noise=true , address=77.90.31.16:4765 , piece_id=6Z63YOOPGEH6JMXSVDR6KIAWZO7WXHOUTZXXWTZYEZVA5YDGNZSQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613341622 (2h3m7.394275006s) [noise=true , address=104.36.82.47:4542 , piece_id=7VATHDYL4G52ITO43MWNWPNL2X62AVSFVY765AX4B2AG5QCIHFLQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613671486 (2h3m7.394604868s) [noise=true , address=79.127.201.215:4343 , piece_id=LZSZ5QJGMGHRKHEQ24GVFLH2VZHAQZUTRV5VVCWO6SRV74JPZJSA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613972571 (2h3m7.394905955s) [noise=true , address=104.36.82.54:4160 , piece_id=KLHY5NZ27H6R46SNVSOBHYH5G4CZ5GZFK2KTXZONX5PGPHZB4KTQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614275533 (2h3m7.395208921s) [noise=true , address=45.91.116.14:4206 , piece_id=U4GASVAEQZ7KP2LWZRANHZ4VEP5L3R2DE4GDURKB5766RYYHNXDA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614572378 (2h3m7.395505757s) [noise=true , address=79.127.220.101:4041 , piece_id=ANJNVQL6CGNZPDWTY3ZDZODH3EP35V7C7PK5LZZ65FJEJLJT322A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614869152 (2h3m7.395802533s) [noise=true , address=79.127.219.38:4015 , piece_id=RPQLDOSQMT7SNTRUQMPPEBMEDAKGN3P4KQCKPNEGS6HFUAL77WZA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614936265 (2h3m7.395869656s) [noise=true , address=77.90.31.12:4620 , piece_id=XRP2ZHRMGUKHSMOQIGBXDJNOJF76EG3O3JENBOCERG34T2GBTSZA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.615461794 (2h3m7.396395185s) [noise=true , address=156.146.43.225:4434 , piece_id=CH2BF7FUDX35B56YEZY2HZBAG3ZLBXJZGIWSN7FCRNGWWWLIPLXA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.615945360 (2h3m7.396878746s) [noise=true , address=104.36.82.46:4510 , piece_id=EQOXPZZWQ33QSVFLX6IKL6RIHFLSVB2KBTE36DSVJNSNTZMJHFSA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.616461681 (2h3m7.397395068s) [noise=true , address=104.36.82.52:4106 , piece_id=4ZZSQQ7CQS3ZG7BLH7Y2WHGWI6GFPTIYBYSAIJXXNZ6TTRGZ74AQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.616899932 (2h3m7.397833318s) [noise=true , address=104.36.82.55:4225 , piece_id=A7FHVFHEE6IGOUUSZTOOLIIZQFWPMDANOZMBSZK2G5G4OUYDGIIA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.617354692 (2h3m7.398288079s) [noise=true , address=77.90.31.13:4656 , piece_id=6JB5DZVWXCYOITFCTJ5C5QMUO4X5WDOTXYB2RHT2KSIFFL7L742A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.699422778 (2h3m7.480356141s) [upload=1886]: Next returned nil segment Jul 26 23:54:51.699445471 (2h3m7.480378858s) [upload=1886]: Waiting for error group managing segments... Jul 26 23:54:51.720281143 (2h3m7.501214521s) [noise=true , address=79.127.219.38:4015 , piece_id=RPQLDOSQMT7SNTRUQMPPEBMEDAKGN3P4KQCKPNEGS6HFUAL77WZA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:51.735111155 (2h3m7.516044525s) [noise=true , address=77.90.31.16:4765 , piece_id=6Z63YOOPGEH6JMXSVDR6KIAWZO7WXHOUTZXXWTZYEZVA5YDGNZSQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:52.117537914 (2h3m7.898471292s) [noise=true , address=156.146.43.225:4434 , piece_id=CH2BF7FUDX35B56YEZY2HZBAG3ZLBXJZGIWSN7FCRNGWWWLIPLXA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:52.208655786 (2h3m7.989589169s) [noise=true , address=79.127.205.245:4249 , piece_id=2GGDQGAWBTBUOQLUPLZU7FKD4WS7AKL6LNNOV5HCHE7RENRURQFA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:52.217561394 (2h3m7.998494757s) [noise=true , address=79.127.205.246:4278 , piece_id=LX2ENB6LMVMG2ZUKG32ZV2WZ5JKZPKHMCK6T62WBDRC73LFV6I6A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:53.480493157 (2h3m9.261426529s) [noise=true , address=77.90.31.12:4620 , piece_id=XRP2ZHRMGUKHSMOQIGBXDJNOJF76EG3O3JENBOCERG34T2GBTSZA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:54.314282140 (2h3m10.095215507s) [noise=true , address=79.127.201.214:4321 , piece_id=PEQ4RK3LDREIGED2ZZFUPRE57XH4KD3VHHKAW7C7FHYCZXEQW3TQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:54.958830009 (2h3m10.739763388s) [noise=true , address=45.91.116.14:4206 , piece_id=U4GASVAEQZ7KP2LWZRANHZ4VEP5L3R2DE4GDURKB5766RYYHNXDA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:59.695538751 (2h3m15.476472128s) [noise=true , address=79.127.220.101:4041 , piece_id=ANJNVQL6CGNZPDWTY3ZDZODH3EP35V7C7PK5LZZ65FJEJLJT322A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:18.516918083 (2h3m34.297851459s) [noise=true , address=79.127.201.215:4343 , piece_id=LZSZ5QJGMGHRKHEQ24GVFLH2VZHAQZUTRV5VVCWO6SRV74JPZJSA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:32.037831589 (2h3m47.818764959s) [noise=true , address=77.90.31.13:4656 , piece_id=6JB5DZVWXCYOITFCTJ5C5QMUO4X5WDOTXYB2RHT2KSIFFL7L742A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:51.546685461 (2h4m7.327618837s) [noise=true , address=104.36.82.47:4542 , piece_id=7VATHDYL4G52ITO43MWNWPNL2X62AVSFVY765AX4B2AG5QCIHFLQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547561359 (2h4m7.328494735s) [noise=true , address=104.36.82.53:4155 , piece_id=XN5QZ2S5UI32H4C6X7E6WOSCDS3WBNZ5DKDTKR2N2O4DHUKH4MPQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547651946 (2h4m7.328585323s) [noise=true , address=104.36.82.54:4160 , piece_id=KLHY5NZ27H6R46SNVSOBHYH5G4CZ5GZFK2KTXZONX5PGPHZB4KTQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: rpc: dial tcp 104.36.82.54:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547721911 (2h4m7.328655276s) [noise=true , address=77.90.31.17:4780 , piece_id=MXK24WZJGMPX3THAEEJDCVCGBB2PX3TBFSM6MQ5H43VUV3RN243A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: write tcp 159.223.143.39:55998->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547789476 (2h4m7.328722851s) [noise=true , address=104.36.82.55:4225 , piece_id=A7FHVFHEE6IGOUUSZTOOLIIZQFWPMDANOZMBSZK2G5G4OUYDGIIA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547876105 (2h4m7.328809484s) [noise=true , address=104.36.82.46:4510 , piece_id=EQOXPZZWQ33QSVFLX6IKL6RIHFLSVB2KBTE36DSVJNSNTZMJHFSA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: rpc: dial tcp 104.36.82.46:4510: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547958673 (2h4m7.328892048s) [noise=true , address=104.36.82.52:4106 , piece_id=4ZZSQQ7CQS3ZG7BLH7Y2WHGWI6GFPTIYBYSAIJXXNZ6TTRGZ74AQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:55:51.547975170 (2h4m7.32890854s) [seg_pos={0 0}, upload=1886]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:55:51.578843242 (2h4m7.359776623s) [upload=1886]: Done uploading segments... Jul 26 23:55:51.578899484 (2h4m7.359832864s) [upload=1886]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.034470426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 30.026066375s FAIL (2025-07-26 23:55:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 6 (2h3m24.604934395s) [upload=1891]: Starting upload Jul 26 23:55:08.824212724 (2h3m24.605146104s) [upload=1891]: Done starting upload Jul 26 23:55:08.824473693 (2h3m24.605407061s) [upload=1891]: Uploading segments... Jul 26 23:55:08.824676568 (2h3m24.605609945s) [upload=1891]: Got next segment. Inline: false Jul 26 23:55:08.824730302 (2h3m24.605663678s) [upload=1891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:08.824736974 (2h3m24.605670351s) [upload=1891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:08.859095743 (2h3m24.640029129s) [seg_pos={0 0}, upload=1891]: Begin upload segment... Jul 26 23:55:08.859160476 (2h3m24.640093846s) [seg_pos={0 0}, upload=1891]: Done begin upload segment. Jul 26 23:55:08.859191679 (2h3m24.640125061s) [noise=true , address=45.91.116.7:4011 , piece_id=776YHSW7RYC3ZVH4RPR62PRNE2UXNTUXMXLGTK5SVMDF5MJYZ5IQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859215605 (2h3m24.640148988s) [noise=true , address=77.90.31.18:4806 , piece_id=VTS3CX7EZZGS726C5BROQOONCGYVTELIM7W6CWSNNV5CAFMJTTNQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859693577 (2h3m24.640626967s) [noise=true , address=77.90.31.11:4553 , piece_id=ENYC7UKSC6FUGGEPG6JXOTD34LSVKD4KVCBTQTRRF2KWM5XR6VLQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859727885 (2h3m24.640661274s) [noise=true , address=77.90.31.17:4770 , piece_id=SAZU4DIFGJ2U67ARSW3KQUSE43N6VY4AIHTM6FQIYSVV4225IVIA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860231888 (2h3m24.641165278s) [noise=true , address=104.36.82.11:4115 , piece_id=JNQPG3KE32EULC4SWTHMAKQ3BDRTTQNXBHP2CTYHBVTXF5T3V23Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860387364 (2h3m24.641320751s) [noise=true , address=77.90.31.12:4592 , piece_id=VCZHG7DNVFTC2TORJDOAWJFYRJL5CQY7765EOPIA57I2SDV2O56Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860638196 (2h3m24.641571582s) [noise=true , address=77.90.31.21:4878 , piece_id=7Y3ZGAVPQHAJBLBYXKMYQYMZ7VJHCKNR2HOMDMNOOXCUBNXJRCNA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860884870 (2h3m24.641818256s) [noise=true , address=104.36.82.7:4910 , piece_id=4HAELEVHC4VBMHYS5ZUA222EZ5YZDEAVXJSKMBXO2ZVQSEHZYAEA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861014206 (2h3m24.641947585s) [noise=true , address=45.91.116.4:4939 , piece_id=6RE4PR4QNPUY5I75IQSV4A7AXBIFSYFD6VAVS3V5EXD5UOP7R5ZA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861309940 (2h3m24.642243322s) [noise=true , address=45.91.116.14:4189 , piece_id=SYK4K3YLSID5OEGIWEXYB2PYZMWPWHVS5IVM6I67CUZIQ5SDYYAQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861351351 (2h3m24.642284748s) [noise=true , address=45.91.116.7:4009 , piece_id=22FKJTPEWIPAW2S4NXW4OS3IJVTNO42SZRNYYUXTM33SJH2HCN6A, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861671962 (2h3m24.642605345s) [noise=true , address=104.36.82.8:4972 , piece_id=QX6VEOPKVL2YP4VG54LEQWAS4XXYXLSTTRUENYH2HYIZJ2D2VEQA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861732627 (2h3m24.642666015s) [noise=true , address=77.90.31.18:4805 , piece_id=VCZO5OWKY2FBL6O5MERCGZYASNLEM7F4OUWNGB5T5TLHCSVE2LCA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862026106 (2h3m24.642959494s) [noise=true , address=45.91.116.8:4048 , piece_id=XT44CG3QN7BSAMUGIT5DJJKQSSTSKGP2BVE653VILJC6EYDEGIFA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862169178 (2h3m24.643102556s) [noise=true , address=45.91.116.5:4298 , piece_id=VXXOTCYFMQ5YRFRTK3P7B42HZCMTRROXFHVEAN4LVQTWDLLLEVKQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862622641 (2h3m24.643556028s) [noise=true , address=104.36.82.4:4803 , piece_id=HH3UBWRB22HFXVGZY2MZK4WT4TV5ZECSOBTYWDTCNDBMVS3U2K5Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.951686160 (2h3m24.73261954s) [upload=1891]: Next returned nil segment Jul 26 23:55:08.951706185 (2h3m24.732639572s) [upload=1891]: Waiting for error group managing segments... Jul 26 23:55:08.955755607 (2h3m24.736688988s) [noise=true , address=45.91.116.4:4939 , piece_id=6RE4PR4QNPUY5I75IQSV4A7AXBIFSYFD6VAVS3V5EXD5UOP7R5ZA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.956456438 (2h3m24.737389801s) [noise=true , address=45.91.116.7:4011 , piece_id=776YHSW7RYC3ZVH4RPR62PRNE2UXNTUXMXLGTK5SVMDF5MJYZ5IQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.957551868 (2h3m24.738485241s) [noise=true , address=45.91.116.8:4048 , piece_id=XT44CG3QN7BSAMUGIT5DJJKQSSTSKGP2BVE653VILJC6EYDEGIFA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.958143228 (2h3m24.739076605s) [noise=true , address=45.91.116.5:4298 , piece_id=VXXOTCYFMQ5YRFRTK3P7B42HZCMTRROXFHVEAN4LVQTWDLLLEVKQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.958670840 (2h3m24.739604205s) [noise=true , address=45.91.116.7:4009 , piece_id=22FKJTPEWIPAW2S4NXW4OS3IJVTNO42SZRNYYUXTM33SJH2HCN6A, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.959966163 (2h3m24.740899531s) [noise=true , address=77.90.31.11:4553 , piece_id=ENYC7UKSC6FUGGEPG6JXOTD34LSVKD4KVCBTQTRRF2KWM5XR6VLQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.987506174 (2h3m24.768439551s) [noise=true , address=45.91.116.14:4189 , piece_id=SYK4K3YLSID5OEGIWEXYB2PYZMWPWHVS5IVM6I67CUZIQ5SDYYAQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:11.029206379 (2h3m26.810139755s) [noise=true , address=77.90.31.12:4592 , piece_id=VCZHG7DNVFTC2TORJDOAWJFYRJL5CQY7765EOPIA57I2SDV2O56Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:38.792571971 (2h3m54.573505332s) [noise=true , address=104.36.82.8:4972 , piece_id=QX6VEOPKVL2YP4VG54LEQWAS4XXYXLSTTRUENYH2HYIZJ2D2VEQA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792681069 (2h3m54.573614445s) [noise=true , address=104.36.82.4:4803 , piece_id=HH3UBWRB22HFXVGZY2MZK4WT4TV5ZECSOBTYWDTCNDBMVS3U2K5Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:53848->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792697299 (2h3m54.573630683s) [noise=true , address=104.36.82.11:4115 , piece_id=JNQPG3KE32EULC4SWTHMAKQ3BDRTTQNXBHP2CTYHBVTXF5T3V23Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:41412->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792746088 (2h3m54.573679457s) [noise=true , address=77.90.31.18:4806 , piece_id=VTS3CX7EZZGS726C5BROQOONCGYVTELIM7W6CWSNNV5CAFMJTTNQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792785701 (2h3m54.57371908s) [noise=true , address=77.90.31.18:4805 , piece_id=VCZO5OWKY2FBL6O5MERCGZYASNLEM7F4OUWNGB5T5TLHCSVE2LCA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:59332->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792800710 (2h3m54.573734077s) [noise=true , address=77.90.31.17:4770 , piece_id=SAZU4DIFGJ2U67ARSW3KQUSE43N6VY4AIHTM6FQIYSVV4225IVIA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 159.223.143.39:35684->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792835255 (2h3m54.573768623s) [noise=true , address=77.90.31.21:4878 , piece_id=7Y3ZGAVPQHAJBLBYXKMYQYMZ7VJHCKNR2HOMDMNOOXCUBNXJRCNA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:34806->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792850658 (2h3m54.573784028s) [noise=true , address=104.36.82.7:4910 , piece_id=4HAELEVHC4VBMHYS5ZUA222EZ5YZDEAVXJSKMBXO2ZVQSEHZYAEA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:41434->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792887535 (2h3m54.57382091s) [seg_pos={0 0}, upload=1891]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:55:38.815750378 (2h3m54.596683745s) [upload=1891]: Done uploading segments... Jul 26 23:55:38.815795701 (2h3m54.596729078s) [upload=1891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026097447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXX] duration: 30.035577391s FAIL (2025-07-26 23:55:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.035607545s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XX.X...X..X.......X...X....XX..............X.X........X...X] duration: 1m0.000834361s FAIL (2025-07-26 23:55:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000850015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 30.0310172s FAIL (2025-07-26 23:55:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== fXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:39970->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:15.582954316 (2h2m31.363887705s) [seg_pos={0 0}, upload=1867]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:54:15.619601382 (2h2m31.400534761s) [upload=1867]: Done uploading segments... Jul 26 23:54:15.619630767 (2h2m31.400564142s) [upload=1867]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:54:38.729043852 (2h2m54.509977238s) [upload=1883]: Starting upload Jul 26 23:54:38.729140031 (2h2m54.510073408s) [upload=1883]: Done starting upload Jul 26 23:54:38.729287878 (2h2m54.510221246s) [upload=1883]: Uploading segments... Jul 26 23:54:38.729568530 (2h2m54.510501901s) [upload=1883]: Got next segment. Inline: false Jul 26 23:54:38.729622997 (2h2m54.510556394s) [upload=1883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:38.729630029 (2h2m54.510563415s) [upload=1883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:38.764743853 (2h2m54.545677232s) [seg_pos={0 0}, upload=1883]: Begin upload segment... Jul 26 23:54:38.764784800 (2h2m54.54571817s) [seg_pos={0 0}, upload=1883]: Done begin upload segment. Jul 26 23:54:38.764808331 (2h2m54.54574171s) [noise=true , address=45.91.116.10:4081 , piece_id=OJWTTTYTU4FXWKB4WJN4FRRT3Z26R6DL4PPRA2AH4JXSOM2BUHKA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.764861679 (2h2m54.54579507s) [noise=true , address=104.36.82.12:4150 , piece_id=4JMF7AN7SVRXHM7S57Z6TZC5WGNBC3WJS44AAO7HCG47PHXGJJUQ, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.765162662 (2h2m54.546096044s) [noise=true , address=45.91.116.9:4261 , piece_id=YO6B3ND5N5X5OXGUNB4JIX55FFDCLN24ZJPM6UKIBOU7FFXJ5C7Q, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.765372897 (2h2m54.546306277s) [noise=true , address=104.36.82.9:4005 , piece_id=VULUSPCYVXY3MUZO2GHSUAMKBMETC7TYRDFA5LCWJAYM5GTFD7ZA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.765510462 (2h2m54.546443842s) [noise=true , address=77.90.31.14:4662 , piece_id=LZMLEHHM2ULSOQAZW6I6PXQ37J3USRSD4JYJOI64E6767E7QK4IA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.765785675 (2h2m54.54671905s) [noise=true , address=104.36.82.2:4731 , piece_id=DHH5VYN4TSFJIVRPGBYM5XO2N24INUJRLIS4KIKHD5FGJLVGBWDA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.765831990 (2h2m54.546765369s) [noise=true , address=104.36.82.10:4079 , piece_id=DUFHOJPQPRSLH6LTDLCNPZEVFDWF6C5G7I6TL2SMHD6O3YPZ4JVQ, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.766166397 (2h2m54.547099772s) [noise=true , address=104.36.82.2:4730 , piece_id=C3Y2ZJUJKATVD7J5DVCLPE2WFOHSUSKLSISWC22REGRWWF7KTIUA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.766189265 (2h2m54.547122641s) [noise=true , address=77.90.31.10:4519 , piece_id=BMLCXXBUJWCG6FMLIV32MGRE2F6QTJBQZFUQHV6BXBPVCT4A6JBA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.766574534 (2h2m54.547507909s) [noise=true , address=45.91.116.12:4227 , piece_id=T4RII7YEHAS7XFHV526LMRYD4DN3JAN5JPFLTCKSE5FRK2BBGJNQ, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.766593574 (2h2m54.547526954s) [noise=true , address=77.90.31.14:4661 , piece_id=36YN65QO2GOY3XIKDNAERJLSWG3VSZDTKOXV4DL4AOPTPE4NAARQ, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.766893845 (2h2m54.54782722s) [noise=true , address=45.91.116.4:4938 , piece_id=HGGWYOI4EUT4WQ5ZFOUB3724PM5TYMM6YU46ZWS2IDEWNWEUCENA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.767240125 (2h2m54.548173506s) [noise=true , address=45.91.116.7:4011 , piece_id=OYCGEZB2BSUGP3TMNIPK7GDBZBENIJFZ3YY7OT3KYDX7JXRK6ESA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.767613956 (2h2m54.548547335s) [noise=true , address=45.91.116.8:4047 , piece_id=K72RIMALBKNVVVX25UEGH6EDYD3TYSOV6ZOETZB5ZHFPOB4GSNTQ, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.768379969 (2h2m54.549313368s) [noise=true , address=45.91.116.5:4298 , piece_id=XTWWHVKWFIJPNUVPT27KTNSKSVVYZVAX3XA654RDSUI3DYQII33A, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.768774722 (2h2m54.549708096s) [noise=true , address=77.90.31.18:4805 , piece_id=VTZ565VYCP63ULEFJ4ZBOQKRQZO74KZVOS3KUYVTKVNDA4BI2NFA, seg_pos={0 0}, upload=1883]: Uploading piece... Jul 26 23:54:38.858394989 (2h2m54.639328367s) [upload=1883]: Next returned nil segment Jul 26 23:54:38.859109450 (2h2m54.640042845s) [upload=1883]: Waiting for error group managing segments... Jul 26 23:54:38.862921037 (2h2m54.643854421s) [noise=true , address=45.91.116.4:4938 , piece_id=HGGWYOI4EUT4WQ5ZFOUB3724PM5TYMM6YU46ZWS2IDEWNWEUCENA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.864661905 (2h2m54.645595285s) [noise=true , address=45.91.116.10:4081 , piece_id=OJWTTTYTU4FXWKB4WJN4FRRT3Z26R6DL4PPRA2AH4JXSOM2BUHKA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.865198317 (2h2m54.646131695s) [noise=true , address=77.90.31.14:4661 , piece_id=36YN65QO2GOY3XIKDNAERJLSWG3VSZDTKOXV4DL4AOPTPE4NAARQ, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.865699720 (2h2m54.646633103s) [noise=true , address=45.91.116.5:4298 , piece_id=XTWWHVKWFIJPNUVPT27KTNSKSVVYZVAX3XA654RDSUI3DYQII33A, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.866236668 (2h2m54.647170048s) [noise=true , address=45.91.116.7:4011 , piece_id=OYCGEZB2BSUGP3TMNIPK7GDBZBENIJFZ3YY7OT3KYDX7JXRK6ESA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.866761730 (2h2m54.647695107s) [noise=true , address=45.91.116.12:4227 , piece_id=T4RII7YEHAS7XFHV526LMRYD4DN3JAN5JPFLTCKSE5FRK2BBGJNQ, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.867708683 (2h2m54.648642046s) [noise=true , address=45.91.116.8:4047 , piece_id=K72RIMALBKNVVVX25UEGH6EDYD3TYSOV6ZOETZB5ZHFPOB4GSNTQ, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.868207778 (2h2m54.649141142s) [noise=true , address=77.90.31.14:4662 , piece_id=LZMLEHHM2ULSOQAZW6I6PXQ37J3USRSD4JYJOI64E6767E7QK4IA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:38.975189214 (2h2m54.756122585s) [noise=true , address=45.91.116.9:4261 , piece_id=YO6B3ND5N5X5OXGUNB4JIX55FFDCLN24ZJPM6UKIBOU7FFXJ5C7Q, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:54:40.358585136 (2h2m56.139518512s) [noise=true , address=77.90.31.10:4519 , piece_id=BMLCXXBUJWCG6FMLIV32MGRE2F6QTJBQZFUQHV6BXBPVCT4A6JBA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: <nil> Jul 26 23:55:08.691532744 (2h3m24.472466126s) [noise=true , address=104.36.82.12:4150 , piece_id=4JMF7AN7SVRXHM7S57Z6TZC5WGNBC3WJS44AAO7HCG47PHXGJJUQ, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691638359 (2h3m24.472571737s) [noise=true , address=104.36.82.2:4730 , piece_id=C3Y2ZJUJKATVD7J5DVCLPE2WFOHSUSKLSISWC22REGRWWF7KTIUA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39240->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691740131 (2h3m24.472673494s) [noise=true , address=104.36.82.2:4731 , piece_id=DHH5VYN4TSFJIVRPGBYM5XO2N24INUJRLIS4KIKHD5FGJLVGBWDA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691802318 (2h3m24.472735695s) [noise=true , address=77.90.31.18:4805 , piece_id=VTZ565VYCP63ULEFJ4ZBOQKRQZO74KZVOS3KUYVTKVNDA4BI2NFA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691865907 (2h3m24.47279927s) [noise=true , address=104.36.82.9:4005 , piece_id=VULUSPCYVXY3MUZO2GHSUAMKBMETC7TYRDFA5LCWJAYM5GTFD7ZA, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691903398 (2h3m24.472836777s) [noise=true , address=104.36.82.10:4079 , piece_id=DUFHOJPQPRSLH6LTDLCNPZEVFDWF6C5G7I6TL2SMHD6O3YPZ4JVQ, seg_pos={0 0}, upload=1883]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:34958->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:08.691949920 (2h3m24.472883305s) [seg_pos={0 0}, upload=1883]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:55:08.720909868 (2h3m24.501843253s) [upload=1883]: Done uploading segments... Jul 26 23:55:08.720941627 (2h3m24.501875012s) [upload=1883]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031042561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032009979s FAIL (2025-07-26 23:54:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 51.504812080 (2h2m7.285745467s) [seg_pos={0 0}, upload=1869]: Begin upload segment... Jul 26 23:53:51.504875029 (2h2m7.285808407s) [seg_pos={0 0}, upload=1869]: Done begin upload segment. Jul 26 23:53:51.504902397 (2h2m7.285835778s) [noise=true , address=79.127.205.245:4248 , piece_id=A5POD4EK2VOR6O5TTMEDP5MSKZDS5IQBXGHJGYXKSLZFHUOCH7SA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.504920249 (2h2m7.285853632s) [noise=true , address=79.127.205.233:4646 , piece_id=WPZAPKB36CF62AX7XHXNZMIC3VZJ2KX4UVLDWEBOMM5PIPYBLGAA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.504986719 (2h2m7.285920099s) [noise=true , address=104.36.82.47:4538 , piece_id=2EFYYCOXLOZIQNAV4YTWU7DVJLOOB33UBK2L5AOYDKMVFXVIG4RQ, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.505312397 (2h2m7.286245763s) [noise=true , address=79.127.205.234:4616 , piece_id=DEDZMALXRXJ63HGHUULGPU2UZR3CPJKUKYRSHUWPF3I57F2CZH4Q, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.505456498 (2h2m7.28638988s) [noise=true , address=77.90.31.10:4544 , piece_id=FBBJRB4B6JPV2D2OUSI5NEOP2X5BKRTAXACCCJRLKEDIH5U5CT5A, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.505940643 (2h2m7.286874026s) [noise=true , address=79.127.205.246:4272 , piece_id=3MUA33AC2SXMTNRB5HSZSHLHMS3UQNXTQ2FUKV4YCNJXZIWNBJVQ, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.506371196 (2h2m7.287304574s) [noise=true , address=104.36.82.55:4198 , piece_id=6A43PLQ44XLHCZLBR66LK32FB4MOTH7N6A7KWY5SIT6BCVINP75A, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.506805981 (2h2m7.287739367s) [noise=true , address=104.36.82.12:4156 , piece_id=FKAK4BZ6WOOZ2Z2HMN3D3GJPCTZK42O6BZD7527W6MHBSIFZKWGA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.507120900 (2h2m7.288054278s) [noise=true , address=79.127.219.38:4018 , piece_id=SIMNDT6LE5AFPQBDEYCLRJSSLWGRKZ6GBYIRQ53YTIOFVYGZL3JQ, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.507479335 (2h2m7.28841271s) [noise=true , address=156.146.43.225:4434 , piece_id=JGEA6O4CTOLGZORNHVOOEGXAAMACBO2BPYKCLQUQIFG3SLLKC35Q, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.507766712 (2h2m7.28870009s) [noise=true , address=104.36.82.54:4190 , piece_id=LXQNQEOTA523BJHH22BHE4ELZHNIHFEI2AKER7OTFXMIPYNZPBIA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.508125644 (2h2m7.289059029s) [noise=true , address=104.36.82.46:4504 , piece_id=KZIIOFZE5K7K5TMF5LTPR2NEC5GT4BAI3AFLGX24G54HDVKMMBMA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.508452239 (2h2m7.289385617s) [noise=true , address=45.91.116.14:4206 , piece_id=VFBVY7HI7DNMWVO5LWQZD3RD7VKUMHYPU4MMN6LSPKVXOYYLZNLQ, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.508832604 (2h2m7.289765994s) [noise=true , address=45.91.116.9:4277 , piece_id=YDFGNT4JHQ7MKKVJFH3KSQWRWUU4VZCABFFFAQU5ZZTBTHVB27OA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.509285263 (2h2m7.290218648s) [noise=true , address=77.90.31.18:4837 , piece_id=VF5FSHUQBUVIPTYJTWVVJ2A53KKIRTBONIWNABSRSRBKERALA7AA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.509715882 (2h2m7.290649267s) [noise=true , address=79.127.201.215:4351 , piece_id=UVGYMOXENG2HLL5Z6UPHLQQALPJUNG2NBO7WQDKSTSOBEEE25MNA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.509749182 (2h2m7.290682566s) [noise=true , address=79.127.205.235:4678 , piece_id=X2FNAB7LNCMULQ75PFKTVYQGZ5O3PPORI2OXRCSWD4QVC63C77XA, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.510216936 (2h2m7.291150322s) [noise=true , address=104.36.82.53:4151 , piece_id=Y623VCPKTS2BLKHV7FG3WIZ6JTVV4VJKAMOP633HFDPNE3JBQVQQ, seg_pos={0 0}, upload=1869]: Uploading piece... Jul 26 23:53:51.578125103 (2h2m7.359058489s) [upload=1869]: Next returned nil segment Jul 26 23:53:51.578144766 (2h2m7.359078151s) [upload=1869]: Waiting for error group managing segments... Jul 26 23:53:51.582085914 (2h2m7.363019284s) [noise=true , address=45.91.116.9:4277 , piece_id=YDFGNT4JHQ7MKKVJFH3KSQWRWUU4VZCABFFFAQU5ZZTBTHVB27OA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:51.648651412 (2h2m7.429584781s) [noise=true , address=77.90.31.10:4544 , piece_id=FBBJRB4B6JPV2D2OUSI5NEOP2X5BKRTAXACCCJRLKEDIH5U5CT5A, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.008652095 (2h2m7.789585476s) [noise=true , address=79.127.219.38:4018 , piece_id=SIMNDT6LE5AFPQBDEYCLRJSSLWGRKZ6GBYIRQ53YTIOFVYGZL3JQ, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.097781565 (2h2m7.878714934s) [noise=true , address=79.127.205.234:4616 , piece_id=DEDZMALXRXJ63HGHUULGPU2UZR3CPJKUKYRSHUWPF3I57F2CZH4Q, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.101799753 (2h2m7.882733141s) [noise=true , address=79.127.205.235:4678 , piece_id=X2FNAB7LNCMULQ75PFKTVYQGZ5O3PPORI2OXRCSWD4QVC63C77XA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.120415800 (2h2m7.901349162s) [noise=true , address=79.127.205.245:4248 , piece_id=A5POD4EK2VOR6O5TTMEDP5MSKZDS5IQBXGHJGYXKSLZFHUOCH7SA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.217477542 (2h2m7.998410921s) [noise=true , address=79.127.205.246:4272 , piece_id=3MUA33AC2SXMTNRB5HSZSHLHMS3UQNXTQ2FUKV4YCNJXZIWNBJVQ, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:52.457313646 (2h2m8.238247014s) [noise=true , address=79.127.205.233:4646 , piece_id=WPZAPKB36CF62AX7XHXNZMIC3VZJ2KX4UVLDWEBOMM5PIPYBLGAA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:53:53.561699111 (2h2m9.342632477s) [noise=true , address=45.91.116.14:4206 , piece_id=VFBVY7HI7DNMWVO5LWQZD3RD7VKUMHYPU4MMN6LSPKVXOYYLZNLQ, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:53:54.607698891 (2h2m10.388632262s) [noise=true , address=156.146.43.225:4434 , piece_id=JGEA6O4CTOLGZORNHVOOEGXAAMACBO2BPYKCLQUQIFG3SLLKC35Q, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:54:04.503507638 (2h2m20.284441014s) [noise=true , address=79.127.201.215:4351 , piece_id=UVGYMOXENG2HLL5Z6UPHLQQALPJUNG2NBO7WQDKSTSOBEEE25MNA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: <nil> Jul 26 23:54:51.437746109 (2h3m7.218679475s) [noise=true , address=104.36.82.53:4151 , piece_id=Y623VCPKTS2BLKHV7FG3WIZ6JTVV4VJKAMOP633HFDPNE3JBQVQQ, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:196fMe3fbxKaPgNs1zQx7Kg3Rh68s6Gtp7mbu3hn654WCbZctx, address:104.36.82.53:4151): protocol: rpc: dial tcp 104.36.82.53:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.437819228 (2h3m7.218752604s) [noise=true , address=104.36.82.54:4190 , piece_id=LXQNQEOTA523BJHH22BHE4ELZHNIHFEI2AKER7OTFXMIPYNZPBIA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.437877969 (2h3m7.218811347s) [noise=true , address=104.36.82.47:4538 , piece_id=2EFYYCOXLOZIQNAV4YTWU7DVJLOOB33UBK2L5AOYDKMVFXVIG4RQ, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.437925548 (2h3m7.218858914s) [noise=true , address=104.36.82.12:4156 , piece_id=FKAK4BZ6WOOZ2Z2HMN3D3GJPCTZK42O6BZD7527W6MHBSIFZKWGA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:104.36.82.12:4156): protocol: rpc: dial tcp 104.36.82.12:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.437983959 (2h3m7.218917338s) [noise=true , address=77.90.31.18:4837 , piece_id=VF5FSHUQBUVIPTYJTWVVJ2A53KKIRTBONIWNABSRSRBKERALA7AA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: rpc: dial tcp 77.90.31.18:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.438059466 (2h3m7.218992842s) [noise=true , address=104.36.82.46:4504 , piece_id=KZIIOFZE5K7K5TMF5LTPR2NEC5GT4BAI3AFLGX24G54HDVKMMBMA, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.438141733 (2h3m7.219075108s) [noise=true , address=104.36.82.55:4198 , piece_id=6A43PLQ44XLHCZLBR66LK32FB4MOTH7N6A7KWY5SIT6BCVINP75A, seg_pos={0 0}, upload=1869]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:33748->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:51.438164007 (2h3m7.219097406s) [seg_pos={0 0}, upload=1869]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:54:51.467445583 (2h3m7.248378959s) [upload=1869]: Done uploading segments... Jul 26 23:54:51.467473378 (2h3m7.248406753s) [upload=1869]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.032030451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX.X.X.X.......XXX...XX.....X.XXX...............X..X.....] duration: 2.451375547s FAIL (2025-07-26 23:54:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.451400493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XX] duration: 30.028111151s FAIL (2025-07-26 23:54:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.028139885s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..XX.X...X..X.......X...X....XX..............X.X........X...] duration: 48.646971615s FAIL (2025-07-26 23:54:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.646994093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 30.038956457s FAIL (2025-07-26 23:54:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917699325 (2h1m39.6986327s) [noise=true , address=104.36.82.3:4765 , piece_id=ETJI7RVECVZYB25I47W6PVSU7EIYNVXLSFW63AB3SJOBEESEVPAA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:35752->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917729439 (2h1m39.698662805s) [seg_pos={0 0}, upload=1855]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:53:23.943930961 (2h1m39.724864331s) [upload=1855]: Done uploading segments... Jul 26 23:53:23.943971452 (2h1m39.724904828s) [upload=1855]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:53:45.616390638 (2h2m1.397324036s) [upload=1867]: Starting upload Jul 26 23:53:45.616495236 (2h2m1.397428617s) [upload=1867]: Done starting upload Jul 26 23:53:45.616627002 (2h2m1.39756037s) [upload=1867]: Uploading segments... Jul 26 23:53:45.617100293 (2h2m1.398033676s) [upload=1867]: Got next segment. Inline: false Jul 26 23:53:45.617152933 (2h2m1.398086314s) [upload=1867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:53:45.617173860 (2h2m1.398107239s) [upload=1867]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:53:45.649803971 (2h2m1.430737352s) [seg_pos={0 0}, upload=1867]: Begin upload segment... Jul 26 23:53:45.649852831 (2h2m1.430786197s) [seg_pos={0 0}, upload=1867]: Done begin upload segment. Jul 26 23:53:45.650027911 (2h2m1.430961279s) [noise=true , address=104.36.82.12:4149 , piece_id=K7SH5BH7VCJJ4UUGEYML3JVIJTDNO2OG5WUA55J3K2GNEBJBV7JA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.650099091 (2h2m1.431032477s) [noise=true , address=45.91.116.6:4975 , piece_id=6ASIPEA7PR36BQELOC546HZ4K37L55ET3WZWDCOJ3NQJEVQJJWJA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.651608751 (2h2m1.432542117s) [noise=true , address=45.91.116.7:4009 , piece_id=O5MKNIH265LGE5HAXMAHOE66FLUNEORSV2DIYC7JTBHGCZROC2IA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.651953762 (2h2m1.43288713s) [noise=true , address=45.91.116.6:4973 , piece_id=EPRUD2DE6RQ7YC7O6XV6WJUEIPMONW3GGMICCXJJVGOXAORZJGVQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.652326633 (2h2m1.433260001s) [noise=true , address=77.90.31.21:4879 , piece_id=NAOAJOOS26WZNEBILZV4KT76I5W6E3NSGI4JLSOGB3UH4ZPCJHDA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.652677847 (2h2m1.43361122s) [noise=true , address=77.90.31.10:4517 , piece_id=2LUXJTMEBEQLICAFXIPVAJDCTV4X63ET7JER6I7S5P3BF4ZZSMUQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.653073032 (2h2m1.4340064s) [noise=true , address=45.91.116.7:4011 , piece_id=OEG3CW3CUAOPNYO3KLLD4OVULPXJHAAQFFUJU2FVIZUUTTVNEONQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.653381988 (2h2m1.43431536s) [noise=true , address=45.91.116.14:4189 , piece_id=SFNZ4Z7TKDZBND7NOAI4REJE7NU35LV3TITJTPGA6OEPEYHGNBBQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.653662846 (2h2m1.434596223s) [noise=true , address=77.90.31.18:4805 , piece_id=V56DG3TF4TC6WUEV3LX6N3AUTD5SY2VPI3JB626TN3PG7OSQHQNA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.654067657 (2h2m1.435001031s) [noise=true , address=45.91.116.11:4117 , piece_id=Q5SAANLVWGFNJCST5VEBUEML7BYXAXPD7ZVTQU3YJFVT2LI7GX2A, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.654373244 (2h2m1.435306611s) [noise=true , address=45.91.116.12:4226 , piece_id=A7ZXUNPY3MDXTZWPQ2Q7E3G3B3GHB72RTKCQ2VUFVNLKBTTOHOBQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.654717042 (2h2m1.435650411s) [noise=true , address=104.36.82.6:4876 , piece_id=N2UIZZAT6RNSIZJDCWFC2ABSAGAOE7HF3EOE5FGJLR5S37TBHORA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.654964212 (2h2m1.435897582s) [noise=true , address=45.91.116.4:4937 , piece_id=DWEQGHDDKBUCUYMSWOXG2WBPKGGIS4LI4UTAOINUNBWA7IHQCMGQ, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.655270969 (2h2m1.436204336s) [noise=true , address=77.90.31.16:4735 , piece_id=WSJLJSIWAA3S5VW4C52WT6T6E5SPCFAUGPBR6LBBT6C77M6FFASA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.655627143 (2h2m1.43656051s) [noise=true , address=104.36.82.7:4910 , piece_id=4HE4662GP6ZU5QLWQ3RKB5MYRP4HDU7NJ4JSMNQUH45A2RFQOS4A, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.655959017 (2h2m1.436892386s) [noise=true , address=45.91.116.9:4262 , piece_id=E76G4ESU42AA6OOGHHXM3I2CSYTPDB23KARHEWG73RQ3NKORL6TA, seg_pos={0 0}, upload=1867]: Uploading piece... Jul 26 23:53:45.739834373 (2h2m1.520767741s) [upload=1867]: Next returned nil segment Jul 26 23:53:45.739849068 (2h2m1.520782436s) [upload=1867]: Waiting for error group managing segments... Jul 26 23:53:45.743879030 (2h2m1.524812409s) [noise=true , address=45.91.116.6:4975 , piece_id=6ASIPEA7PR36BQELOC546HZ4K37L55ET3WZWDCOJ3NQJEVQJJWJA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.744789788 (2h2m1.525723156s) [noise=true , address=77.90.31.10:4517 , piece_id=2LUXJTMEBEQLICAFXIPVAJDCTV4X63ET7JER6I7S5P3BF4ZZSMUQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.745594919 (2h2m1.526528296s) [noise=true , address=45.91.116.6:4973 , piece_id=EPRUD2DE6RQ7YC7O6XV6WJUEIPMONW3GGMICCXJJVGOXAORZJGVQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.746223211 (2h2m1.527156573s) [noise=true , address=45.91.116.4:4937 , piece_id=DWEQGHDDKBUCUYMSWOXG2WBPKGGIS4LI4UTAOINUNBWA7IHQCMGQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.746307330 (2h2m1.527240706s) [noise=true , address=45.91.116.7:4009 , piece_id=O5MKNIH265LGE5HAXMAHOE66FLUNEORSV2DIYC7JTBHGCZROC2IA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.746769855 (2h2m1.52770322s) [noise=true , address=45.91.116.12:4226 , piece_id=A7ZXUNPY3MDXTZWPQ2Q7E3G3B3GHB72RTKCQ2VUFVNLKBTTOHOBQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.746812194 (2h2m1.527745564s) [noise=true , address=45.91.116.11:4117 , piece_id=Q5SAANLVWGFNJCST5VEBUEML7BYXAXPD7ZVTQU3YJFVT2LI7GX2A, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.746912945 (2h2m1.527846306s) [noise=true , address=77.90.31.16:4735 , piece_id=WSJLJSIWAA3S5VW4C52WT6T6E5SPCFAUGPBR6LBBT6C77M6FFASA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.747354513 (2h2m1.528287892s) [noise=true , address=45.91.116.7:4011 , piece_id=OEG3CW3CUAOPNYO3KLLD4OVULPXJHAAQFFUJU2FVIZUUTTVNEONQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.791712579 (2h2m1.572645949s) [noise=true , address=45.91.116.9:4262 , piece_id=E76G4ESU42AA6OOGHHXM3I2CSYTPDB23KARHEWG73RQ3NKORL6TA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:45.881263912 (2h2m1.662197291s) [noise=true , address=45.91.116.14:4189 , piece_id=SFNZ4Z7TKDZBND7NOAI4REJE7NU35LV3TITJTPGA6OEPEYHGNBBQ, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:53:47.142758059 (2h2m2.923691427s) [noise=true , address=77.90.31.21:4879 , piece_id=NAOAJOOS26WZNEBILZV4KT76I5W6E3NSGI4JLSOGB3UH4ZPCJHDA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: <nil> Jul 26 23:54:15.582670645 (2h2m31.363604021s) [noise=true , address=77.90.31.18:4805 , piece_id=V56DG3TF4TC6WUEV3LX6N3AUTD5SY2VPI3JB626TN3PG7OSQHQNA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:33338->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:15.582757719 (2h2m31.363691084s) [noise=true , address=104.36.82.7:4910 , piece_id=4HE4662GP6ZU5QLWQ3RKB5MYRP4HDU7NJ4JSMNQUH45A2RFQOS4A, seg_pos={0 0}, upload=1867]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; context deadline exceeded Jul 26 23:54:15.582838144 (2h2m31.363771519s) [noise=true , address=104.36.82.12:4149 , piece_id=K7SH5BH7VCJJ4UUGEYML3JVIJTDNO2OG5WUA55J3K2GNEBJBV7JA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:15.582901647 (2h2m31.363835023s) [noise=true , address=104.36.82.6:4876 , piece_id=N2UIZZAT6RNSIZJDCWFC2ABSAGAOE7HF3EOE5FGJLR5S37TBHORA, seg_pos={0 0}, upload=1867]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:39970->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:54:15.582954316 (2h2m31.363887705s) [seg_pos={0 0}, upload=1867]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:54:15.619601382 (2h2m31.400534761s) [upload=1867]: Done uploading segments... Jul 26 23:54:15.619630767 (2h2m31.400564142s) [upload=1867]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.038981033s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XX.X.X.X.......XXX...XX.....X.XXX...............X..X....] duration: 1.928851569s FAIL (2025-07-26 23:54:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.928879723s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [.XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033195324s FAIL (2025-07-26 23:53:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=1854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:52:51.346923639 (2h1m7.127857022s) [upload=1854]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:52:51.381689484 (2h1m7.162622849s) [seg_pos={0 0}, upload=1854]: Begin upload segment... Jul 26 23:52:51.381744641 (2h1m7.162678006s) [seg_pos={0 0}, upload=1854]: Done begin upload segment. Jul 26 23:52:51.381780770 (2h1m7.162714151s) [noise=true , address=79.127.220.101:4048 , piece_id=UI2K3JPLJFNDWMTVLM4FS5VDKZNPZPDAUPPMP2KUZHGXFOQEPUSQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.381806683 (2h1m7.162740069s) [noise=true , address=104.36.82.47:4550 , piece_id=OSJIBY673YTHX5Y7KYDVQLTBYGBHC4KBGWSHF3R2AKCSJBTKPYUA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.382156971 (2h1m7.163090353s) [noise=true , address=79.127.205.246:4280 , piece_id=NA3NK4EWXZUVUNHGZD5BPFTFP3FFF4N6S6JWZULFUDIXEQZ7O2RQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.382447191 (2h1m7.163380578s) [noise=true , address=104.36.82.10:4090 , piece_id=BITNK5372ETBETBVKJ5QI7BKFO5YJO4FZA7YOMTV5YRIFGQNYQ3A, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.382503326 (2h1m7.163436704s) [noise=true , address=79.127.205.235:4680 , piece_id=YFUNLPO5EZDMX3NPJQD3A76IBR6MFLMVX3PEUAI3LRIEFRXW4YMA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.382807521 (2h1m7.163740956s) [noise=true , address=77.90.31.13:4644 , piece_id=AX6GPTTIHLSZS5IFXJ2D6DONQPBRT2MY4ROPAE27KYRFKGPK2J4Q, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.383012576 (2h1m7.163945961s) [noise=true , address=79.127.219.38:4006 , piece_id=KXU5VN5LOSN2NKGHSAVOE6W6GXGRWBWN52QT3E7OLEF3QHN4L6WA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.383150966 (2h1m7.164084345s) [noise=true , address=104.36.82.55:4200 , piece_id=UUYRCI6K7V6JDY6GZIKVR25X3LZQ4KSNFWHY3OIXVH63IZ56BYQQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.383523259 (2h1m7.164456644s) [noise=true , address=156.146.43.225:4449 , piece_id=VPXHOE63OH2ZMPRDQBRIKUJCZWGSVEPY3TLXICLPGY6V6WSTVBXA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.383892121 (2h1m7.164825505s) [noise=true , address=77.90.31.12:4608 , piece_id=UIVEPEPE2ZIAG7XKB5HIJPF5X6KF4WQZM7WHJDH362FQ3TLRWEMQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.384277115 (2h1m7.165210498s) [noise=true , address=79.127.205.234:4604 , piece_id=F5LFJG2RREP7PU2DERHHTH37R4I4FO2PUBN4COESZKOCNGOLMP5Q, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.384618406 (2h1m7.165551805s) [noise=true , address=79.127.220.101:4041 , piece_id=WT4LECPO5BEYHUR3BI4QYRRUI4WHBY25ISTIWXIULVGB7Q4FOE4A, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.384794776 (2h1m7.165728169s) [noise=true , address=79.127.219.47:4048 , piece_id=T3FSLYCXFOVVYNXI3W6IJBZM4GOKVCOWIBMTMPF7FYWLTAB7GGGQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.384991171 (2h1m7.165924563s) [noise=true , address=104.36.82.53:4142 , piece_id=QRXHHFVXZ7J7ONMNFZVIBQUKOJHGS5ARI5THJHUAVWR42K6MB4QA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.385335441 (2h1m7.166268827s) [noise=true , address=79.127.205.245:4249 , piece_id=YQM37GQXVGWWTWSJVJGS66I2GJ4ZY62NLMXKELX7QUELHPZOXGPQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.385686873 (2h1m7.166620254s) [noise=true , address=104.36.82.56:4250 , piece_id=Z5YQKCJ3NKA2X5IHFUBW6VNDA2TVCZB34QJO5MV5OKIOD3XQDZSQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.385740765 (2h1m7.166674147s) [noise=true , address=104.36.82.12:4152 , piece_id=TNB7VSKOTH2SAKRXVRTYR2O7Y636YCQ7JYEKRKXKAMKRQV7PQJHQ, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.386098654 (2h1m7.167032032s) [noise=true , address=104.36.82.46:4500 , piece_id=ZIES4BWBVU632K4JH4TYOKQOPRBJXHG4WMWJ6ZI5R63E3QV3C4YA, seg_pos={0 0}, upload=1854]: Uploading piece... Jul 26 23:52:51.451419013 (2h1m7.232352399s) [upload=1854]: Next returned nil segment Jul 26 23:52:51.451439107 (2h1m7.232372491s) [upload=1854]: Waiting for error group managing segments... Jul 26 23:52:51.485361384 (2h1m7.266294764s) [noise=true , address=79.127.219.38:4006 , piece_id=KXU5VN5LOSN2NKGHSAVOE6W6GXGRWBWN52QT3E7OLEF3QHN4L6WA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:51.952417544 (2h1m7.733350906s) [noise=true , address=79.127.205.235:4680 , piece_id=YFUNLPO5EZDMX3NPJQD3A76IBR6MFLMVX3PEUAI3LRIEFRXW4YMA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:51.980401153 (2h1m7.761334532s) [noise=true , address=79.127.205.234:4604 , piece_id=F5LFJG2RREP7PU2DERHHTH37R4I4FO2PUBN4COESZKOCNGOLMP5Q, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:52.157135649 (2h1m7.938069011s) [noise=true , address=79.127.205.246:4280 , piece_id=NA3NK4EWXZUVUNHGZD5BPFTFP3FFF4N6S6JWZULFUDIXEQZ7O2RQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:52.383980608 (2h1m8.164913977s) [noise=true , address=79.127.220.101:4048 , piece_id=UI2K3JPLJFNDWMTVLM4FS5VDKZNPZPDAUPPMP2KUZHGXFOQEPUSQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:52.740695348 (2h1m8.521628717s) [noise=true , address=79.127.205.245:4249 , piece_id=YQM37GQXVGWWTWSJVJGS66I2GJ4ZY62NLMXKELX7QUELHPZOXGPQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:55.124303624 (2h1m10.905236993s) [noise=true , address=79.127.220.101:4041 , piece_id=WT4LECPO5BEYHUR3BI4QYRRUI4WHBY25ISTIWXIULVGB7Q4FOE4A, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:56.457386885 (2h1m12.238320261s) [noise=true , address=77.90.31.12:4608 , piece_id=UIVEPEPE2ZIAG7XKB5HIJPF5X6KF4WQZM7WHJDH362FQ3TLRWEMQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:56.554618480 (2h1m12.335551851s) [noise=true , address=156.146.43.225:4449 , piece_id=VPXHOE63OH2ZMPRDQBRIKUJCZWGSVEPY3TLXICLPGY6V6WSTVBXA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:52:58.848743495 (2h1m14.629676858s) [noise=true , address=79.127.219.47:4048 , piece_id=T3FSLYCXFOVVYNXI3W6IJBZM4GOKVCOWIBMTMPF7FYWLTAB7GGGQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:53:33.309673443 (2h1m49.09060681s) [noise=true , address=77.90.31.13:4644 , piece_id=AX6GPTTIHLSZS5IFXJ2D6DONQPBRT2MY4ROPAE27KYRFKGPK2J4Q, seg_pos={0 0}, upload=1854]: Done uploading piece. err: <nil> Jul 26 23:53:51.317419032 (2h2m7.098352408s) [noise=true , address=104.36.82.56:4250 , piece_id=Z5YQKCJ3NKA2X5IHFUBW6VNDA2TVCZB34QJO5MV5OKIOD3XQDZSQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:12oHdA7jYDKSYwWLB975bJ3Fu27xjfZDC9utpnQpszmYjVz2CdD, address:104.36.82.56:4250): protocol: rpc: dial tcp 104.36.82.56:4250: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317561404 (2h2m7.098494785s) [noise=true , address=104.36.82.53:4142 , piece_id=QRXHHFVXZ7J7ONMNFZVIBQUKOJHGS5ARI5THJHUAVWR42K6MB4QA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317616487 (2h2m7.098549862s) [noise=true , address=104.36.82.12:4152 , piece_id=TNB7VSKOTH2SAKRXVRTYR2O7Y636YCQ7JYEKRKXKAMKRQV7PQJHQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: rpc: dial tcp 104.36.82.12:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317650049 (2h2m7.098583424s) [noise=true , address=104.36.82.46:4500 , piece_id=ZIES4BWBVU632K4JH4TYOKQOPRBJXHG4WMWJ6ZI5R63E3QV3C4YA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317694306 (2h2m7.098627681s) [noise=true , address=104.36.82.47:4550 , piece_id=OSJIBY673YTHX5Y7KYDVQLTBYGBHC4KBGWSHF3R2AKCSJBTKPYUA, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317730412 (2h2m7.098663786s) [noise=true , address=104.36.82.10:4090 , piece_id=BITNK5372ETBETBVKJ5QI7BKFO5YJO4FZA7YOMTV5YRIFGQNYQ3A, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: rpc: dial tcp 104.36.82.10:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317774447 (2h2m7.098707822s) [noise=true , address=104.36.82.55:4200 , piece_id=UUYRCI6K7V6JDY6GZIKVR25X3LZQ4KSNFWHY3OIXVH63IZ56BYQQ, seg_pos={0 0}, upload=1854]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:51.317790287 (2h2m7.098723649s) [seg_pos={0 0}, upload=1854]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:53:51.347840256 (2h2m7.128773623s) [upload=1854]: Done uploading segments... Jul 26 23:53:51.347863040 (2h2m7.128796408s) [upload=1854]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.033212581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.X] duration: 30.02499286s FAIL (2025-07-26 23:53:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.025013535s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X..XX.X...X..X.......X...X....XX..............X.X........X..] duration: 43.906165603s FAIL (2025-07-26 23:53:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.90619464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X...] duration: 2.033249647s FAIL (2025-07-26 23:53:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.033282862s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 30.029703365s FAIL (2025-07-26 23:53:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== d enough pieces (needed at least 13 but got 7) Jul 26 23:52:53.957375956 (2h1m9.738309319s) [upload=1855]: Starting upload Jul 26 23:52:53.957445216 (2h1m9.738378597s) [upload=1855]: Done starting upload Jul 26 23:52:53.957622573 (2h1m9.738555948s) [upload=1855]: Uploading segments... Jul 26 23:52:53.957906679 (2h1m9.738840041s) [upload=1855]: Got next segment. Inline: false Jul 26 23:52:53.957977419 (2h1m9.738910798s) [upload=1855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:52:53.957984620 (2h1m9.738917998s) [upload=1855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:52:53.997767998 (2h1m9.778701374s) [seg_pos={0 0}, upload=1855]: Begin upload segment... Jul 26 23:52:53.997878652 (2h1m9.77881204s) [noise=true , address=104.36.82.7:4909 , piece_id=X2S2DKTAY5UGNNZHJ3YWQFOJUOVNOMSZINTG42NE6S7AQH4WK4VA, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:53.998423642 (2h1m9.779357027s) [noise=true , address=45.91.116.8:4048 , piece_id=V6VB2SH4IWD6BKHQXXAL3E43A4V6S6GKLPTLQXVER3XTS66MS7RQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:53.999288046 (2h1m9.780221428s) [seg_pos={0 0}, upload=1855]: Done begin upload segment. Jul 26 23:52:53.999364491 (2h1m9.780297876s) [noise=true , address=104.36.82.8:4972 , piece_id=2EBOIE3TU4LU7CQOJCJ6RMKXHIIHM3KUSF32663XEF3K2BU62ENQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:53.999909729 (2h1m9.780843108s) [noise=true , address=77.90.31.12:4592 , piece_id=VQUNPIOBGI5Z7ESNM3MAOPN5KO7DXVA4JZB6VKXWZZF7XMF65H2Q, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.000584928 (2h1m9.781518337s) [noise=true , address=104.36.82.3:4765 , piece_id=ETJI7RVECVZYB25I47W6PVSU7EIYNVXLSFW63AB3SJOBEESEVPAA, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.001155973 (2h1m9.78208935s) [noise=true , address=45.91.116.12:4227 , piece_id=G3FZKVEIFLUKLKT62LFSQDASWUD4RAEJBH633HSESOPU4DRHPRUA, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.001683817 (2h1m9.782617192s) [noise=true , address=45.91.116.7:4009 , piece_id=6H5OQYGCW7TNGYWCN7LGV3LVRZCMBOB3IECOEWGECGO5WZ67WYOQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.002131252 (2h1m9.783064637s) [noise=true , address=77.90.31.21:4879 , piece_id=UL6NRR4OY7553U5HCDDLTCACCXHEBFQ72CXSNPXP5UZ4LF775AFQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.002682736 (2h1m9.783616121s) [noise=true , address=45.91.116.5:4298 , piece_id=VW5IHLY2VBOZEWHYBZVCNDMVZU2KHZ7C2R3K5W4CDSMQHJOHCO5Q, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.003390501 (2h1m9.784323906s) [noise=true , address=104.36.82.15:4259 , piece_id=HII4KT3YXI6DYRGT2ERXRLYWYAOUXYB6ADRPUWUDEKPWUMJOQRSQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.003808084 (2h1m9.784741465s) [noise=true , address=104.36.82.14:4222 , piece_id=7EOEPUI4WMWFN2TS7KIY7XA4PPKZIKAXVYPNHHMB7D3B2VRMZYMA, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.004234028 (2h1m9.785167415s) [noise=true , address=45.91.116.9:4261 , piece_id=HFAUU7FWB25PGKG6X7H34PXPQTQC7SOIADBQNUATOZ2ZUI2ZRA3Q, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.004672062 (2h1m9.785605447s) [noise=true , address=104.36.82.5:4839 , piece_id=GJTX6Y7QMHTOASG4CRC4WB557PZTT37C7TAPTQDL4XBDOPAJUGIQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.005079605 (2h1m9.786012981s) [noise=true , address=104.36.82.15:4258 , piece_id=Z6XHOEDGT45PK5SRIBCU7GRHAVSQQ47EFFCDI5EEYI46MHHY6XLA, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.005503091 (2h1m9.786436479s) [noise=true , address=45.91.116.10:4081 , piece_id=QNB3I3C54CVGB5PWYJRCCWUT5ZHCPORGKWYVV3KCHYBNFEFZWEIQ, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.005870505 (2h1m9.786803917s) [noise=true , address=104.36.82.11:4115 , piece_id=BGOAZJ46KX7UCMULOT3VL3QIWJI3HMMZLOHHLPFIAA4J5ZXGZT2Q, seg_pos={0 0}, upload=1855]: Uploading piece... Jul 26 23:52:54.099907804 (2h1m9.880841192s) [upload=1855]: Next returned nil segment Jul 26 23:52:54.099955676 (2h1m9.880889053s) [upload=1855]: Waiting for error group managing segments... Jul 26 23:52:54.104637439 (2h1m9.885570827s) [noise=true , address=45.91.116.7:4009 , piece_id=6H5OQYGCW7TNGYWCN7LGV3LVRZCMBOB3IECOEWGECGO5WZ67WYOQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.105005852 (2h1m9.885939215s) [noise=true , address=45.91.116.12:4227 , piece_id=G3FZKVEIFLUKLKT62LFSQDASWUD4RAEJBH633HSESOPU4DRHPRUA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.105269631 (2h1m9.886203008s) [noise=true , address=45.91.116.10:4081 , piece_id=QNB3I3C54CVGB5PWYJRCCWUT5ZHCPORGKWYVV3KCHYBNFEFZWEIQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.105649691 (2h1m9.886583069s) [noise=true , address=45.91.116.8:4048 , piece_id=V6VB2SH4IWD6BKHQXXAL3E43A4V6S6GKLPTLQXVER3XTS66MS7RQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.105805410 (2h1m9.886738794s) [noise=true , address=77.90.31.21:4879 , piece_id=UL6NRR4OY7553U5HCDDLTCACCXHEBFQ72CXSNPXP5UZ4LF775AFQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.109690844 (2h1m9.89062421s) [noise=true , address=45.91.116.5:4298 , piece_id=VW5IHLY2VBOZEWHYBZVCNDMVZU2KHZ7C2R3K5W4CDSMQHJOHCO5Q, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:54.534065668 (2h1m10.31499905s) [noise=true , address=45.91.116.9:4261 , piece_id=HFAUU7FWB25PGKG6X7H34PXPQTQC7SOIADBQNUATOZ2ZUI2ZRA3Q, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:52:55.581277077 (2h1m11.362210442s) [noise=true , address=77.90.31.12:4592 , piece_id=VQUNPIOBGI5Z7ESNM3MAOPN5KO7DXVA4JZB6VKXWZZF7XMF65H2Q, seg_pos={0 0}, upload=1855]: Done uploading piece. err: <nil> Jul 26 23:53:23.917255547 (2h1m39.69818893s) [noise=true , address=104.36.82.7:4909 , piece_id=X2S2DKTAY5UGNNZHJ3YWQFOJUOVNOMSZINTG42NE6S7AQH4WK4VA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917337886 (2h1m39.698271265s) [noise=true , address=104.36.82.15:4259 , piece_id=HII4KT3YXI6DYRGT2ERXRLYWYAOUXYB6ADRPUWUDEKPWUMJOQRSQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917464538 (2h1m39.698397913s) [noise=true , address=104.36.82.14:4222 , piece_id=7EOEPUI4WMWFN2TS7KIY7XA4PPKZIKAXVYPNHHMB7D3B2VRMZYMA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:46136->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917531621 (2h1m39.698464997s) [noise=true , address=104.36.82.5:4839 , piece_id=GJTX6Y7QMHTOASG4CRC4WB557PZTT37C7TAPTQDL4XBDOPAJUGIQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:46068->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917570098 (2h1m39.698503472s) [noise=true , address=104.36.82.15:4258 , piece_id=Z6XHOEDGT45PK5SRIBCU7GRHAVSQQ47EFFCDI5EEYI46MHHY6XLA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:56840->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917629442 (2h1m39.698562817s) [noise=true , address=104.36.82.11:4115 , piece_id=BGOAZJ46KX7UCMULOT3VL3QIWJI3HMMZLOHHLPFIAA4J5ZXGZT2Q, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:60930->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917672996 (2h1m39.698606371s) [noise=true , address=104.36.82.8:4972 , piece_id=2EBOIE3TU4LU7CQOJCJ6RMKXHIIHM3KUSF32663XEF3K2BU62ENQ, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917699325 (2h1m39.6986327s) [noise=true , address=104.36.82.3:4765 , piece_id=ETJI7RVECVZYB25I47W6PVSU7EIYNVXLSFW63AB3SJOBEESEVPAA, seg_pos={0 0}, upload=1855]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:35752->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:53:23.917729439 (2h1m39.698662805s) [seg_pos={0 0}, upload=1855]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:53:23.943930961 (2h1m39.724864331s) [upload=1855]: Done uploading segments... Jul 26 23:53:23.943971452 (2h1m39.724904828s) [upload=1855]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.029746843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [..XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029641221s FAIL (2025-07-26 23:52:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== g_pos={0 0}, upload=1841]: Begin upload segment... Jul 26 23:51:51.268143050 (2h0m7.049076431s) [seg_pos={0 0}, upload=1841]: Done begin upload segment. Jul 26 23:51:51.268315631 (2h0m7.049249019s) [noise=true , address=77.90.31.18:4831 , piece_id=TWZEVU2JDGTJNG6TEKUBAPEMFZWQDGMYVFHTZDJOBNMLCV7VPS7Q, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.268376459 (2h0m7.049309848s) [noise=true , address=104.36.82.12:4154 , piece_id=KCRWXZQKG5IPKGYQEDRPXRPIJJZIXDTU6PD6BNMPZJRQFANKE4AQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.268855721 (2h0m7.049789111s) [noise=true , address=64.49.13.138:4483 , piece_id=OWVU5XCPEUYZ2TVD4INLPBS6MDWWOWULLFTT7ZHBS2JFHAKC5TTQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.269820619 (2h0m7.050754006s) [noise=true , address=79.127.219.38:4009 , piece_id=4RKS2OVG3OVXHRL4NQGZ5P7DZGC7LX77CSU2XE6NOHVGRYP3FIEQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.270557410 (2h0m7.051490798s) [noise=true , address=77.90.31.11:4558 , piece_id=UPXLSTCS4JYLF4Y235QBAGJIZ7F3WCWQWIMOAURSZMTHXKCCJBEQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.270988281 (2h0m7.051921669s) [noise=true , address=104.36.82.42:4822 , piece_id=GXSYIZS6LRRWTT43XBSEQYH4FQD4RPDCWQUHKRNSMHQBHDW4YY3Q, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.271771499 (2h0m7.052704887s) [noise=true , address=79.127.205.245:4244 , piece_id=QAAJ34A3R3R3UCNEHFBYBJYZ6YXF7LX5KUXDKQ35YRA2B3FXXABA, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.272481432 (2h0m7.05341482s) [noise=true , address=77.90.31.13:4644 , piece_id=3O6GQINQF5ODFLMWMILN6QW2EKJAGLL5BR4RZJTG35XXABO5DV6Q, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.273205588 (2h0m7.054138974s) [noise=true , address=104.36.82.54:4169 , piece_id=GDNTIV6W7F4SF2QBFNPIQCIYUWBON2I554XGPUL7XTXW5TBOZJ7Q, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.273889357 (2h0m7.054822746s) [noise=true , address=45.91.116.9:4277 , piece_id=5KHMN25GZFSP23PFN7BF3W75FH2NPWK4XR3L6XUBJGCXWQOF4JEQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.274537180 (2h0m7.05547057s) [noise=true , address=104.36.82.49:4989 , piece_id=3EPMTW7PXMXDRN44UD3TRBRMSHMHSQBYZ23NMH6SXC56KD4WFM5Q, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.275337544 (2h0m7.056270941s) [noise=true , address=104.36.82.46:4481 , piece_id=FLOYHAJDZZGUJ7RCV2MVAEULC5NMNGTKJAN32TR64PB52YERBLOQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.277638103 (2h0m7.058571485s) [noise=true , address=79.127.205.246:4262 , piece_id=GZS3R4HIZC2TYN3FREBQ53WJBKSBYNTS6BFCYH6IRAATH653TCMQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.284794278 (2h0m7.065727672s) [noise=true , address=79.127.201.215:4349 , piece_id=MAH6YX6B4YX7ADGICXCXSHDIFBKGZYK3HAQXBXLN6UXGVWIELBMQ, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.288214498 (2h0m7.069147875s) [noise=true , address=79.127.205.235:4682 , piece_id=IIDXKHGXZQMDTHY2OBX3NKI2QVHQDPUHNKAS4KX35EKOWGKRKUKA, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.290917867 (2h0m7.071851253s) [noise=true , address=79.127.205.234:4607 , piece_id=5SQBA6OB4ET6I3TKEF7IOHCSTUFZKRP64HOWCDXVRVW3Q3WS5V3A, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.293810699 (2h0m7.074744084s) [noise=true , address=77.90.31.15:4727 , piece_id=ANVDAVLDX6WNVTY2QEPUZSUFFWBM4TMDWLUN65J3BPW7BTMYN3PA, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.296702506 (2h0m7.077635897s) [noise=true , address=104.36.82.51:4049 , piece_id=FU3QMK6M6TCMEBZVUAE36BEENEZ7HJ2XY47K55DY36ZC23HPDL3A, seg_pos={0 0}, upload=1841]: Uploading piece... Jul 26 23:51:51.354940841 (2h0m7.135874208s) [upload=1841]: Next returned nil segment Jul 26 23:51:51.354960462 (2h0m7.135893839s) [upload=1841]: Waiting for error group managing segments... Jul 26 23:51:51.359113841 (2h0m7.140047223s) [noise=true , address=77.90.31.11:4558 , piece_id=UPXLSTCS4JYLF4Y235QBAGJIZ7F3WCWQWIMOAURSZMTHXKCCJBEQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:51.403206552 (2h0m7.184139929s) [noise=true , address=45.91.116.9:4277 , piece_id=5KHMN25GZFSP23PFN7BF3W75FH2NPWK4XR3L6XUBJGCXWQOF4JEQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:51.865818890 (2h0m7.646752265s) [noise=true , address=79.127.205.235:4682 , piece_id=IIDXKHGXZQMDTHY2OBX3NKI2QVHQDPUHNKAS4KX35EKOWGKRKUKA, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:51.882918726 (2h0m7.663852094s) [noise=true , address=79.127.205.246:4262 , piece_id=GZS3R4HIZC2TYN3FREBQ53WJBKSBYNTS6BFCYH6IRAATH653TCMQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:51.886158492 (2h0m7.667091861s) [noise=true , address=79.127.205.234:4607 , piece_id=5SQBA6OB4ET6I3TKEF7IOHCSTUFZKRP64HOWCDXVRVW3Q3WS5V3A, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:51.962965066 (2h0m7.743898432s) [noise=true , address=64.49.13.138:4483 , piece_id=OWVU5XCPEUYZ2TVD4INLPBS6MDWWOWULLFTT7ZHBS2JFHAKC5TTQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:52.046689331 (2h0m7.827622702s) [noise=true , address=79.127.205.245:4244 , piece_id=QAAJ34A3R3R3UCNEHFBYBJYZ6YXF7LX5KUXDKQ35YRA2B3FXXABA, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:52.403486482 (2h0m8.184419852s) [noise=true , address=79.127.219.38:4009 , piece_id=4RKS2OVG3OVXHRL4NQGZ5P7DZGC7LX77CSU2XE6NOHVGRYP3FIEQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:51:56.550383782 (2h0m12.331317164s) [noise=true , address=77.90.31.13:4644 , piece_id=3O6GQINQF5ODFLMWMILN6QW2EKJAGLL5BR4RZJTG35XXABO5DV6Q, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:52:01.494577921 (2h0m17.275511302s) [noise=true , address=79.127.201.215:4349 , piece_id=MAH6YX6B4YX7ADGICXCXSHDIFBKGZYK3HAQXBXLN6UXGVWIELBMQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:52:07.744932666 (2h0m23.525866031s) [noise=true , address=77.90.31.15:4727 , piece_id=ANVDAVLDX6WNVTY2QEPUZSUFFWBM4TMDWLUN65J3BPW7BTMYN3PA, seg_pos={0 0}, upload=1841]: Done uploading piece. err: <nil> Jul 26 23:52:51.202219101 (2h1m6.983152467s) [noise=true , address=104.36.82.54:4169 , piece_id=GDNTIV6W7F4SF2QBFNPIQCIYUWBON2I554XGPUL7XTXW5TBOZJ7Q, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 159.223.143.39:40690->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202299786 (2h1m6.983233153s) [noise=true , address=77.90.31.18:4831 , piece_id=TWZEVU2JDGTJNG6TEKUBAPEMFZWQDGMYVFHTZDJOBNMLCV7VPS7Q, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 159.223.143.39:55886->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202352924 (2h1m6.983286288s) [noise=true , address=104.36.82.42:4822 , piece_id=GXSYIZS6LRRWTT43XBSEQYH4FQD4RPDCWQUHKRNSMHQBHDW4YY3Q, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:104.36.82.42:4822): protocol: write tcp 159.223.143.39:49280->104.36.82.42:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202361615 (2h1m6.983294997s) [noise=true , address=104.36.82.51:4049 , piece_id=FU3QMK6M6TCMEBZVUAE36BEENEZ7HJ2XY47K55DY36ZC23HPDL3A, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: write tcp 159.223.143.39:33478->104.36.82.51:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202405945 (2h1m6.98333931s) [noise=true , address=104.36.82.12:4154 , piece_id=KCRWXZQKG5IPKGYQEDRPXRPIJJZIXDTU6PD6BNMPZJRQFANKE4AQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: write tcp 159.223.143.39:49358->104.36.82.12:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202447166 (2h1m6.983380533s) [noise=true , address=104.36.82.46:4481 , piece_id=FLOYHAJDZZGUJ7RCV2MVAEULC5NMNGTKJAN32TR64PB52YERBLOQ, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202484852 (2h1m6.983418227s) [noise=true , address=104.36.82.49:4989 , piece_id=3EPMTW7PXMXDRN44UD3TRBRMSHMHSQBYZ23NMH6SXC56KD4WFM5Q, seg_pos={0 0}, upload=1841]: Done uploading piece. err: ecclient: upload failed (node:1boeQC5mQx4MVWhWAJFs3g4DtJMpT1tw4QYrZQ3c2RbDjRAAJc, address:104.36.82.49:4989): protocol: write tcp 159.223.143.39:40406->104.36.82.49:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:51.202527176 (2h1m6.983460561s) [seg_pos={0 0}, upload=1841]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:52:51.230138258 (2h1m7.011071625s) [upload=1841]: Done uploading segments... Jul 26 23:52:51.230163862 (2h1m7.011097225s) [upload=1841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.029662988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 30.034538966s FAIL (2025-07-26 23:52:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3550086 (2h0m23.334483471s) [upload=1845]: Done starting upload Jul 26 23:52:07.553669993 (2h0m23.33460337s) [upload=1845]: Uploading segments... Jul 26 23:52:07.553924023 (2h0m23.334857388s) [upload=1845]: Got next segment. Inline: false Jul 26 23:52:07.553976521 (2h0m23.334909906s) [upload=1845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:52:07.553982603 (2h0m23.334915988s) [upload=1845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:52:07.635497055 (2h0m23.416430441s) [seg_pos={0 0}, upload=1845]: Begin upload segment... Jul 26 23:52:07.635600120 (2h0m23.416533507s) [noise=true , address=45.91.116.14:4190 , piece_id=L3BFZCNZMBZPFQMUBBIOGMHLQWXGGFNW7VZMB2EBWQNFSTSNBQCA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.637086421 (2h0m23.418019814s) [seg_pos={0 0}, upload=1845]: Done begin upload segment. Jul 26 23:52:07.637148008 (2h0m23.418081409s) [noise=true , address=45.91.116.7:4010 , piece_id=FALBDYT6DBXEXPNO54BO3HQ3UIRLR35IWUXB67WGWQ7EXGFRBRSA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.637478107 (2h0m23.418411497s) [noise=true , address=77.90.31.12:4590 , piece_id=I6ORYFJGWJT5V2T3D67A23ZGJEUUG3ZT3IO43PZUUOJVD7XQIEBA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.637633242 (2h0m23.418566627s) [noise=true , address=45.91.116.8:4047 , piece_id=XJD76TY6WSO7HN3C7FIELDCINT3DKRN3NDMTKQF2HFQ57TWOPTKA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.637936334 (2h0m23.418869723s) [noise=true , address=104.36.82.12:4149 , piece_id=LEL22JGXWZVHB77BE3WXSGDNVMH7V2JBBVX5ITYYFBODM3T6SGCQ, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.638045221 (2h0m23.418978607s) [noise=true , address=77.90.31.14:4662 , piece_id=5IY34Z7HHH6CVBIBPMFMOSL4NNGJUKJWUZ7LQDW4AAGXTN7VU2YA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.638449455 (2h0m23.419382837s) [noise=true , address=104.36.82.3:4767 , piece_id=7NIWP7XU3Y5ZNPOGG3SGA7FFI2NOD6PLFBVJRLXXC6QKL3JWAEIQ, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.638485652 (2h0m23.419419041s) [noise=true , address=104.36.82.6:4874 , piece_id=PUOSSFTJ773FEVOLA5VPLP2CKIG4LAECIZXKIXSRMWGC5XLQ7U4A, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.638828108 (2h0m23.419761497s) [noise=true , address=104.36.82.12:4151 , piece_id=UOW6QBIZSSLZBMPMOG436BTPBUHXTUMGFQ7E37ZD2T5XZ4Z5RJCA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.638889293 (2h0m23.419822683s) [noise=true , address=45.91.116.12:4225 , piece_id=XQCPTOGV3KTNSORHGYVVJWFDNAGIR6NLLYBWAHXFYSYNIF37XXSQ, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.639257993 (2h0m23.420191383s) [noise=true , address=104.36.82.16:4293 , piece_id=MYM26CFY44SO7KZR6FHR6SNFTCBB76D44YPRGLUSWSSITDYUIF3Q, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.639316190 (2h0m23.420249571s) [noise=true , address=104.36.82.5:4839 , piece_id=IJX7A4UHZQBLTGYCXOVP5EDJQV6SH4EJWUQQQLDNT7UQRYEP3RWA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.639605370 (2h0m23.420538752s) [noise=true , address=45.91.116.11:4119 , piece_id=A44B4THYUJP6NWKSO2SX7CL22X6MCRFYMMTECDZV4N26L6JGAPRA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.639710628 (2h0m23.420644015s) [noise=true , address=77.90.31.18:4807 , piece_id=EMBOJLMMHXEBO3WXTBJQUXGDMICJELAL65ABE33Z4N562EIQXYJA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.639973271 (2h0m23.420906651s) [noise=true , address=104.36.82.11:4115 , piece_id=6T2Q34GXPCCCRRD5RDTC3XGISBAVJ5RY7WOQH2QIEPSKS4N6K3BA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.640417895 (2h0m23.421351291s) [noise=true , address=104.36.82.9:4007 , piece_id=IVOYX64X4UKTPRHING6ONTFBUOICE3M7UCVE6DKQBQS7FXKTVAYA, seg_pos={0 0}, upload=1845]: Uploading piece... Jul 26 23:52:07.730674270 (2h0m23.511607648s) [upload=1845]: Next returned nil segment Jul 26 23:52:07.730696634 (2h0m23.511630043s) [upload=1845]: Waiting for error group managing segments... Jul 26 23:52:07.740953073 (2h0m23.521886438s) [noise=true , address=45.91.116.12:4225 , piece_id=XQCPTOGV3KTNSORHGYVVJWFDNAGIR6NLLYBWAHXFYSYNIF37XXSQ, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:07.741739276 (2h0m23.522672646s) [noise=true , address=77.90.31.14:4662 , piece_id=5IY34Z7HHH6CVBIBPMFMOSL4NNGJUKJWUZ7LQDW4AAGXTN7VU2YA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:07.742517463 (2h0m23.523450846s) [noise=true , address=45.91.116.11:4119 , piece_id=A44B4THYUJP6NWKSO2SX7CL22X6MCRFYMMTECDZV4N26L6JGAPRA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:07.743222737 (2h0m23.524156111s) [noise=true , address=45.91.116.7:4010 , piece_id=FALBDYT6DBXEXPNO54BO3HQ3UIRLR35IWUXB67WGWQ7EXGFRBRSA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:07.744081859 (2h0m23.525015237s) [noise=true , address=45.91.116.8:4047 , piece_id=XJD76TY6WSO7HN3C7FIELDCINT3DKRN3NDMTKQF2HFQ57TWOPTKA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:07.779151505 (2h0m23.560084866s) [noise=true , address=45.91.116.14:4190 , piece_id=L3BFZCNZMBZPFQMUBBIOGMHLQWXGGFNW7VZMB2EBWQNFSTSNBQCA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:09.141769077 (2h0m24.922702462s) [noise=true , address=77.90.31.12:4590 , piece_id=I6ORYFJGWJT5V2T3D67A23ZGJEUUG3ZT3IO43PZUUOJVD7XQIEBA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: <nil> Jul 26 23:52:37.523411697 (2h0m53.304345072s) [noise=true , address=104.36.82.12:4151 , piece_id=UOW6QBIZSSLZBMPMOG436BTPBUHXTUMGFQ7E37ZD2T5XZ4Z5RJCA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:38894->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.523720036 (2h0m53.304653423s) [noise=true , address=104.36.82.16:4293 , piece_id=MYM26CFY44SO7KZR6FHR6SNFTCBB76D44YPRGLUSWSSITDYUIF3Q, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.523807228 (2h0m53.304740605s) [noise=true , address=77.90.31.18:4807 , piece_id=EMBOJLMMHXEBO3WXTBJQUXGDMICJELAL65ABE33Z4N562EIQXYJA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.523879886 (2h0m53.304813266s) [noise=true , address=104.36.82.5:4839 , piece_id=IJX7A4UHZQBLTGYCXOVP5EDJQV6SH4EJWUQQQLDNT7UQRYEP3RWA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524006320 (2h0m53.304939701s) [noise=true , address=104.36.82.11:4115 , piece_id=6T2Q34GXPCCCRRD5RDTC3XGISBAVJ5RY7WOQH2QIEPSKS4N6K3BA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:37422->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524223900 (2h0m53.30515727s) [noise=true , address=104.36.82.9:4007 , piece_id=IVOYX64X4UKTPRHING6ONTFBUOICE3M7UCVE6DKQBQS7FXKTVAYA, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524411776 (2h0m53.305345157s) [noise=true , address=104.36.82.6:4874 , piece_id=PUOSSFTJ773FEVOLA5VPLP2CKIG4LAECIZXKIXSRMWGC5XLQ7U4A, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524507451 (2h0m53.305440831s) [noise=true , address=104.36.82.12:4149 , piece_id=LEL22JGXWZVHB77BE3WXSGDNVMH7V2JBBVX5ITYYFBODM3T6SGCQ, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524556975 (2h0m53.305490352s) [noise=true , address=104.36.82.3:4767 , piece_id=7NIWP7XU3Y5ZNPOGG3SGA7FFI2NOD6PLFBVJRLXXC6QKL3JWAEIQ, seg_pos={0 0}, upload=1845]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:37.524594980 (2h0m53.305528367s) [seg_pos={0 0}, upload=1845]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:52:37.555485973 (2h0m53.336419366s) [upload=1845]: Done uploading segments... Jul 26 23:52:37.555543367 (2h0m53.336476755s) [upload=1845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.034566944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.] duration: 2.466739847s FAIL (2025-07-26 23:52:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.466875706s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X..XX.X...X..X.......X...X....XX..............X.X........X.] duration: 22.437069796s FAIL (2025-07-26 23:52:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.437100603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X..] duration: 639.521622ms FAIL (2025-07-26 23:52:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 639.542183ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X.] duration: 24.692138661s FAIL (2025-07-26 23:52:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.692159903s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.039936918s FAIL (2025-07-26 23:52:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.039967944s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X..XX.X...X..X.......X...X....XX..............X.X........X] duration: 1m0.001315644s FAIL (2025-07-26 23:52:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001353991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [...XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026904236s FAIL (2025-07-26 23:51:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .959129282s) [seg_pos={0 0}, upload=1827]: Done begin upload segment. Jul 26 23:50:51.178261382 (1h59m6.959194762s) [noise=true , address=104.36.82.56:4243 , piece_id=OF5JCTG3JM4HWRJFEVW7CH6RRYLAXULAYMSDEIJNANK5HFNYHLAA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.179146680 (1h59m6.960080065s) [noise=true , address=79.127.205.234:4610 , piece_id=3TW5UIUDCZXV5VRIQAWSVA4NIGUHWXIRSYUFFZ6WXWS2C33ROHQA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.179203137 (1h59m6.960136514s) [noise=true , address=156.146.43.225:4448 , piece_id=XWSP242FPYUME47HXF72SHNDK7YT6FWNSPD2RIZTR35JEESWMHJA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.179565343 (1h59m6.960498725s) [noise=true , address=77.90.31.11:4569 , piece_id=FLMT4B76OUTAR5XAR6S2FDCPNUTM37IKETF6UBUG7MRGUV2VH75A, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.179884017 (1h59m6.960817394s) [noise=true , address=104.36.82.46:4481 , piece_id=JT3AXWC4H4XZHKQMLW5XHLSDKQWZOPB6V7DVTJ7NLM7E3QQDDOWQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.179914277 (1h59m6.960847652s) [noise=true , address=64.49.13.140:4514 , piece_id=OCC7KSCOS6BE6PEOBUS4Q7MZRAGYDI7CD6Y5MLLO5FFBY7RADHUQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.180443644 (1h59m6.961377028s) [noise=true , address=104.36.82.47:4517 , piece_id=K4B2XNXTV7DPLI3CJD6IUQKL27Y6KWGV22GS4FNAR762K2I2GGYA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.180738898 (1h59m6.961672281s) [noise=true , address=77.90.31.16:4767 , piece_id=JTU7P7QS4WJIOCSBOUDWUOFYL7RIE4MX34HNKMW7CTXBTPHXV5OA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.180876611 (1h59m6.961809999s) [noise=true , address=104.36.82.53:4121 , piece_id=IU2HT47XTNDQPGCCP3KFVWF6VCLX3DVISCZTUTWSAYESUVSM6PLQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.181245114 (1h59m6.962178497s) [noise=true , address=77.90.31.17:4792 , piece_id=653PY5EYIYPAJUSIR62IXOERWFB347DPZHCL5ZAJ6WSP5AHI7NJQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.181407995 (1h59m6.962341373s) [noise=true , address=79.127.205.246:4277 , piece_id=7IMNLDOGS4HJBQK6XTBCYGUIEOPMA7V6L33J6J4P2TEOHVZ6NVWQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.181593110 (1h59m6.962526489s) [noise=true , address=79.127.201.215:4356 , piece_id=7D4IMDRCZ5XFNPK2T7ZHAD3SNQGGVKRSPPQKG5FPXHKO77X2A7RQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.181968126 (1h59m6.962901512s) [noise=true , address=45.91.116.5:4306 , piece_id=V5IURFLBU6H65RGYQZFM23O6QYDXSFL3EXPZC73CM2DGKNQTVRAA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.182043158 (1h59m6.962976545s) [noise=true , address=77.90.31.15:4722 , piece_id=MLP3H6AXPRGNDM6VEWLZ56ORDHMDGDQOVBHVFCDQR7X4XUMGSZ3A, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.182618477 (1h59m6.963551857s) [noise=true , address=64.49.13.138:4482 , piece_id=HGUIJCKFXCMKTRQHINT6UKT4LKWW7NATHX6ULX5Q42GIQ2WZ6E4A, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.183269947 (1h59m6.964203327s) [noise=true , address=104.36.82.55:4222 , piece_id=MVC7DX33LDG3UBNHLBHDUJVUGDHUCH772NDMDZAV5Q23V2PFGFOQ, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.183819898 (1h59m6.964753276s) [noise=true , address=79.127.220.98:4422 , piece_id=R4XQY356LY7N2U65JQFZIK6N4ZLZQVU74Z7HRIFBQAP4NTZ4NTQA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.185669167 (1h59m6.966602543s) [noise=true , address=104.36.82.54:4167 , piece_id=66WKVVW54BKJ4ZWSWFL5UWCL34F2ZVSIS6UQPYZ4H42G6TA5BDOA, seg_pos={0 0}, upload=1827]: Uploading piece... Jul 26 23:50:51.277784191 (1h59m7.058717576s) [upload=1827]: Next returned nil segment Jul 26 23:50:51.277812777 (1h59m7.058746397s) [upload=1827]: Waiting for error group managing segments... Jul 26 23:50:51.308303375 (1h59m7.089236761s) [noise=true , address=77.90.31.11:4569 , piece_id=FLMT4B76OUTAR5XAR6S2FDCPNUTM37IKETF6UBUG7MRGUV2VH75A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:51.702448121 (1h59m7.483381509s) [noise=true , address=45.91.116.5:4306 , piece_id=V5IURFLBU6H65RGYQZFM23O6QYDXSFL3EXPZC73CM2DGKNQTVRAA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:51.746084254 (1h59m7.527017688s) [noise=true , address=79.127.205.246:4277 , piece_id=7IMNLDOGS4HJBQK6XTBCYGUIEOPMA7V6L33J6J4P2TEOHVZ6NVWQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:51.801116499 (1h59m7.582049889s) [noise=true , address=79.127.205.234:4610 , piece_id=3TW5UIUDCZXV5VRIQAWSVA4NIGUHWXIRSYUFFZ6WXWS2C33ROHQA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:51.971134786 (1h59m7.752068149s) [noise=true , address=64.49.13.138:4482 , piece_id=HGUIJCKFXCMKTRQHINT6UKT4LKWW7NATHX6ULX5Q42GIQ2WZ6E4A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:53.733508775 (1h59m9.514442145s) [noise=true , address=79.127.220.98:4422 , piece_id=R4XQY356LY7N2U65JQFZIK6N4ZLZQVU74Z7HRIFBQAP4NTZ4NTQA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:56.067135334 (1h59m11.848068711s) [noise=true , address=156.146.43.225:4448 , piece_id=XWSP242FPYUME47HXF72SHNDK7YT6FWNSPD2RIZTR35JEESWMHJA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:57.227642526 (1h59m13.008575902s) [noise=true , address=64.49.13.140:4514 , piece_id=OCC7KSCOS6BE6PEOBUS4Q7MZRAGYDI7CD6Y5MLLO5FFBY7RADHUQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:50:58.197785065 (1h59m13.978718435s) [noise=true , address=77.90.31.16:4767 , piece_id=JTU7P7QS4WJIOCSBOUDWUOFYL7RIE4MX34HNKMW7CTXBTPHXV5OA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:51:17.923930239 (1h59m33.704863602s) [noise=true , address=79.127.201.215:4356 , piece_id=7D4IMDRCZ5XFNPK2T7ZHAD3SNQGGVKRSPPQKG5FPXHKO77X2A7RQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil> Jul 26 23:51:51.106833575 (2h0m6.88776696s) [noise=true , address=104.36.82.46:4481 , piece_id=JT3AXWC4H4XZHKQMLW5XHLSDKQWZOPB6V7DVTJ7NLM7E3QQDDOWQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.106946732 (2h0m6.887880113s) [noise=true , address=77.90.31.15:4722 , piece_id=MLP3H6AXPRGNDM6VEWLZ56ORDHMDGDQOVBHVFCDQR7X4XUMGSZ3A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:51:51.107112453 (2h0m6.888045842s) [noise=true , address=77.90.31.17:4792 , piece_id=653PY5EYIYPAJUSIR62IXOERWFB347DPZHCL5ZAJ6WSP5AHI7NJQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: rpc: dial tcp 77.90.31.17:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107473667 (2h0m6.888407051s) [noise=true , address=104.36.82.55:4222 , piece_id=MVC7DX33LDG3UBNHLBHDUJVUGDHUCH772NDMDZAV5Q23V2PFGFOQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:104.36.82.55:4222): protocol: rpc: dial tcp 104.36.82.55:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107553162 (2h0m6.88848654s) [noise=true , address=104.36.82.56:4243 , piece_id=OF5JCTG3JM4HWRJFEVW7CH6RRYLAXULAYMSDEIJNANK5HFNYHLAA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: write tcp 159.223.143.39:35570->104.36.82.56:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107649947 (2h0m6.888583313s) [noise=true , address=104.36.82.47:4517 , piece_id=K4B2XNXTV7DPLI3CJD6IUQKL27Y6KWGV22GS4FNAR762K2I2GGYA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:41120->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107743740 (2h0m6.888677116s) [noise=true , address=104.36.82.53:4121 , piece_id=IU2HT47XTNDQPGCCP3KFVWF6VCLX3DVISCZTUTWSAYESUVSM6PLQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107819214 (2h0m6.888752583s) [noise=true , address=104.36.82.54:4167 , piece_id=66WKVVW54BKJ4ZWSWFL5UWCL34F2ZVSIS6UQPYZ4H42G6TA5BDOA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 159.223.143.39:47212->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:51.107858434 (2h0m6.888791823s) [seg_pos={0 0}, upload=1827]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:51:51.131672555 (2h0m6.912605924s) [upload=1827]: Done uploading segments... Jul 26 23:51:51.131700105 (2h0m6.912633482s) [upload=1827]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.02692711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X] duration: 30.001596798s FAIL (2025-07-26 23:51:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001623646s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 30.0296708s FAIL (2025-07-26 23:51:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 26 23:51:16.261048963 (1h59m32.041982357s) [upload=1824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:51:16.377982237 (1h59m32.158915605s) [upload=1832]: Starting upload Jul 26 23:51:16.378077340 (1h59m32.159010729s) [upload=1832]: Done starting upload Jul 26 23:51:16.378466007 (1h59m32.159399385s) [upload=1832]: Uploading segments... Jul 26 23:51:16.378534938 (1h59m32.159468317s) [upload=1832]: Got next segment. Inline: false Jul 26 23:51:16.378568467 (1h59m32.159501843s) [upload=1832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:51:16.378573507 (1h59m32.159506884s) [upload=1832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:51:16.412670039 (1h59m32.193603416s) [seg_pos={0 0}, upload=1832]: Begin upload segment... Jul 26 23:51:16.412732682 (1h59m32.193666047s) [seg_pos={0 0}, upload=1832]: Done begin upload segment. Jul 26 23:51:16.412762000 (1h59m32.193695379s) [noise=true , address=77.90.31.19:4841 , piece_id=BXTC5RAEVKXDPQNZZWWQA3ABWLGEICPT5IGTUQDBIAN4RFYGGKMQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.413212564 (1h59m32.194145943s) [noise=true , address=104.36.82.7:4912 , piece_id=2BGCACSENNV62Q6OC2DMYY3LTAR63AJRR7UYKRJQM2HRRUGWOZSA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.413447576 (1h59m32.194380959s) [noise=true , address=45.91.116.4:4940 , piece_id=H7PFRUIEX3MROPO7LQ2N3X75FC35QIM4CNBB4LFDP5F7PC2T56WA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.414208622 (1h59m32.195142009s) [noise=true , address=77.90.31.21:4878 , piece_id=ZWHN4LMK6L6ENQ7DWOCFXODAOBZABCPHS5PWQXBINMU6RZMSE2PA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.414255649 (1h59m32.195189037s) [noise=true , address=45.91.116.14:4189 , piece_id=7IPZVQZPG4XO3ULSRBXWVAJ5QRJAUXDCVR52G3WISLLKJ7MYRAXQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.414735270 (1h59m32.195668648s) [noise=true , address=77.90.31.18:4805 , piece_id=QQENGNBKMLYC5BVNHIE34LOW3UU5TI52W7E5RUBI5G7S7QH3FY4Q, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.414776049 (1h59m32.195709429s) [noise=true , address=104.36.82.8:4969 , piece_id=LFJKW5Y2HQJ4NHDJ37Z265FB5FQMNNMOLMC5YTZ6MHTXSWRKDNPQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.415149906 (1h59m32.196083291s) [noise=true , address=104.36.82.8:4972 , piece_id=S6XF26HSSBRVJMHHNJ2KULQRUDS4S7FO2AHSKK4YKM5FRLLWFAHA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.415330759 (1h59m32.196264149s) [noise=true , address=104.36.82.16:4294 , piece_id=DVSZEY64WXHTTOKQNNQDY64T4TIZS2C7NIRAFGIYTVDIQEB43UWQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.415622799 (1h59m32.196556187s) [noise=true , address=77.90.31.17:4772 , piece_id=BXCJ2CN5JN345BFIUGHMFXFD7O7AU5BRQS6HIY7PZTP7KO7NTVJA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.415897740 (1h59m32.196831126s) [noise=true , address=104.36.82.15:4257 , piece_id=T2RH4BAFLUIYMX2HHNHIKMPUI7KSYPXYQHD2NYWISOLTCHWPWHQA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.416022039 (1h59m32.196955426s) [noise=true , address=77.90.31.10:4520 , piece_id=TI5OHJ3M5MVUEZRZHOXX3AZ6MYTWW4YJ6SBQGA35YXLEN7F4CWJQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.416530132 (1h59m32.197463521s) [noise=true , address=104.36.82.13:4187 , piece_id=BP2Q373XFGHP5RVWY3H4Y3LNNK3SF4UCMZ6HACYRPTYEFW3JRTOA, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.416697932 (1h59m32.197631309s) [noise=true , address=45.91.116.5:4297 , piece_id=EFLU373IG6SWZSUQGZOYFNUNFQXF45V2CTHYM5I5QX7WEUU7GE5A, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.416910861 (1h59m32.197844249s) [noise=true , address=45.91.116.13:4153 , piece_id=B6WGKDV56QWEUU7PEODDXKUCZOG3SOQG2FBXO2OOI5WTHG3666AQ, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.417152076 (1h59m32.198085462s) [noise=true , address=45.91.116.12:4225 , piece_id=SLXVYTU2BM5CSU56FO6UWFQEY5CH4DKFLQCD3YQQ4R24QFVJ6T4Q, seg_pos={0 0}, upload=1832]: Uploading piece... Jul 26 23:51:16.526433751 (1h59m32.307367145s) [upload=1832]: Next returned nil segment Jul 26 23:51:16.526464196 (1h59m32.307397578s) [upload=1832]: Waiting for error group managing segments... Jul 26 23:51:16.529936697 (1h59m32.310870065s) [noise=true , address=45.91.116.13:4153 , piece_id=B6WGKDV56QWEUU7PEODDXKUCZOG3SOQG2FBXO2OOI5WTHG3666AQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.530001187 (1h59m32.310934559s) [noise=true , address=77.90.31.10:4520 , piece_id=TI5OHJ3M5MVUEZRZHOXX3AZ6MYTWW4YJ6SBQGA35YXLEN7F4CWJQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.530937367 (1h59m32.311870735s) [noise=true , address=77.90.31.21:4878 , piece_id=ZWHN4LMK6L6ENQ7DWOCFXODAOBZABCPHS5PWQXBINMU6RZMSE2PA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.530959441 (1h59m32.311892825s) [noise=true , address=45.91.116.12:4225 , piece_id=SLXVYTU2BM5CSU56FO6UWFQEY5CH4DKFLQCD3YQQ4R24QFVJ6T4Q, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.531544648 (1h59m32.312478019s) [noise=true , address=45.91.116.4:4940 , piece_id=H7PFRUIEX3MROPO7LQ2N3X75FC35QIM4CNBB4LFDP5F7PC2T56WA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.532489779 (1h59m32.313423159s) [noise=true , address=45.91.116.5:4297 , piece_id=EFLU373IG6SWZSUQGZOYFNUNFQXF45V2CTHYM5I5QX7WEUU7GE5A, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:16.569773807 (1h59m32.350707174s) [noise=true , address=45.91.116.14:4189 , piece_id=7IPZVQZPG4XO3ULSRBXWVAJ5QRJAUXDCVR52G3WISLLKJ7MYRAXQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:17.879349361 (1h59m33.660282739s) [noise=true , address=77.90.31.19:4841 , piece_id=BXTC5RAEVKXDPQNZZWWQA3ABWLGEICPT5IGTUQDBIAN4RFYGGKMQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil> Jul 26 23:51:46.348332716 (2h0m2.129266106s) [noise=true , address=104.36.82.15:4257 , piece_id=T2RH4BAFLUIYMX2HHNHIKMPUI7KSYPXYQHD2NYWISOLTCHWPWHQA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348427245 (2h0m2.129360623s) [noise=true , address=77.90.31.17:4772 , piece_id=BXCJ2CN5JN345BFIUGHMFXFD7O7AU5BRQS6HIY7PZTP7KO7NTVJA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348495376 (2h0m2.129428742s) [noise=true , address=104.36.82.13:4187 , piece_id=BP2Q373XFGHP5RVWY3H4Y3LNNK3SF4UCMZ6HACYRPTYEFW3JRTOA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348553293 (2h0m2.129486668s) [noise=true , address=77.90.31.18:4805 , piece_id=QQENGNBKMLYC5BVNHIE34LOW3UU5TI52W7E5RUBI5G7S7QH3FY4Q, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348616454 (2h0m2.129549829s) [noise=true , address=104.36.82.8:4969 , piece_id=LFJKW5Y2HQJ4NHDJ37Z265FB5FQMNNMOLMC5YTZ6MHTXSWRKDNPQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348674420 (2h0m2.129607797s) [noise=true , address=104.36.82.8:4972 , piece_id=S6XF26HSSBRVJMHHNJ2KULQRUDS4S7FO2AHSKK4YKM5FRLLWFAHA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348732514 (2h0m2.129665882s) [noise=true , address=104.36.82.16:4294 , piece_id=DVSZEY64WXHTTOKQNNQDY64T4TIZS2C7NIRAFGIYTVDIQEB43UWQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348787969 (2h0m2.129721345s) [noise=true , address=104.36.82.7:4912 , piece_id=2BGCACSENNV62Q6OC2DMYY3LTAR63AJRR7UYKRJQM2HRRUGWOZSA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:40614->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:46.348809952 (2h0m2.129743338s) [seg_pos={0 0}, upload=1832]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:51:46.375129894 (2h0m2.156063321s) [upload=1832]: Done uploading segments... Jul 26 23:51:46.375165498 (2h0m2.156098876s) [upload=1832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.029689117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 30.028697262s FAIL (2025-07-26 23:51:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:50:14.774834627 (1h58m30.555768005s) [upload=1806]: Done uploading segments... Jul 26 23:50:14.774880812 (1h58m30.555814191s) [upload=1806]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:50:46.263134063 (1h59m2.044067453s) [upload=1824]: Starting upload Jul 26 23:50:46.263216917 (1h59m2.044150291s) [upload=1824]: Done starting upload Jul 26 23:50:46.263526278 (1h59m2.044459648s) [upload=1824]: Uploading segments... Jul 26 23:50:46.263581707 (1h59m2.04451509s) [upload=1824]: Got next segment. Inline: false Jul 26 23:50:46.263621851 (1h59m2.044555235s) [upload=1824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:50:46.263628197 (1h59m2.044561582s) [upload=1824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:50:46.294400796 (1h59m2.075334183s) [seg_pos={0 0}, upload=1824]: Begin upload segment... Jul 26 23:50:46.294464897 (1h59m2.075398281s) [seg_pos={0 0}, upload=1824]: Done begin upload segment. Jul 26 23:50:46.294524059 (1h59m2.075457438s) [noise=true , address=104.36.82.13:4187 , piece_id=NCP7XFGLC4HR3JHAURMNVJ3KGOO5ZIC45ZM7447XBQJSG3ETRRNQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.295059351 (1h59m2.075992734s) [noise=true , address=45.91.116.6:4974 , piece_id=T2HM65HYT7ZMU4U26S3AAEUIZ7W5TCOMQ2I4H4SRAMBSA6SOTBHQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.295467438 (1h59m2.076400815s) [noise=true , address=45.91.116.8:4047 , piece_id=I6A3I7KVZKDNOFC75R5O24RIH2DQ3XHZPID7QVNKH6ZQHG234DVA, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.295861637 (1h59m2.076795017s) [noise=true , address=104.36.82.9:4005 , piece_id=XY4YQ5OI6TOMIZ6ZXDEB4J6VC4P5GRFDLOT7FKIJOYVWJXW6DL6Q, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.296432319 (1h59m2.077365704s) [noise=true , address=45.91.116.6:4975 , piece_id=CVZLNWPBWEJYLT4LBFYCBEKYXYH7NOY7GNJPVZW35E7BUPHCBYPQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.296884733 (1h59m2.077818119s) [noise=true , address=77.90.31.15:4697 , piece_id=OG7L57X66JSNP3DO52E6LTCAE4HCBUK3NBPGIOVUYC6QBLEX5OHQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.297301936 (1h59m2.078235321s) [noise=true , address=104.36.82.16:4294 , piece_id=SGX35GWGTDCM5FIRIGCHEEUOR46DLOU5ZRVMWYN5WVULXDRDGRMQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.297686767 (1h59m2.078620147s) [noise=true , address=104.36.82.9:4007 , piece_id=XH2DW4QTSPULNSQ2K3DAYU737ZH5VXBSSLCRQBZ7YSWC5ZAMUIMQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.298569556 (1h59m2.079502936s) [noise=true , address=104.36.82.5:4839 , piece_id=5RQC5COYAZ5WM3V76UE55QBH3QRCSPNQPX2MT572IOODFHNZ5EYQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.298946623 (1h59m2.079880004s) [noise=true , address=45.91.116.10:4083 , piece_id=NMWGYAVQVMVXABLZXZ7ENAZNTNWS45HEWBFQRXXSSMQVHNC35JPQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.299355342 (1h59m2.080288724s) [noise=true , address=45.91.116.9:4262 , piece_id=M5JDD66PKMFZBKFX2JMXASO5FMR6DMXY6WOLDC3MSLIVPUANI77Q, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.299724032 (1h59m2.080657419s) [noise=true , address=104.36.82.4:4801 , piece_id=A6RPYR5SNYJ3344DBDC4R3RTM3IKQ4DKHJVNFINPI5V6XZD2BVBA, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.300148671 (1h59m2.081082058s) [noise=true , address=45.91.116.4:4938 , piece_id=NV3AVDIDY7KAKY73SS4PM6NW2DVDQWJINKKTKAR4FNGBSJDJIXGQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.300543936 (1h59m2.081477321s) [noise=true , address=77.90.31.13:4628 , piece_id=6OIXFOZSSJHKWFHDCIA55FLKIHCC3BQYR7WTDSLX42Q6226FXM7A, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.300886438 (1h59m2.081819826s) [noise=true , address=104.36.82.2:4732 , piece_id=XMWHE6SDRP7AJZAZVUSXYZQJSR245STM6JPJQL3MSWUNFUF7D7HA, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.301238368 (1h59m2.082171752s) [noise=true , address=45.91.116.14:4189 , piece_id=252XE2CKXYGKMMHCACZHVQ4S6MX7YS6IBTFS3BJRJFSI36O2UOTQ, seg_pos={0 0}, upload=1824]: Uploading piece... Jul 26 23:50:46.385635730 (1h59m2.166569107s) [upload=1824]: Next returned nil segment Jul 26 23:50:46.385662191 (1h59m2.166595576s) [upload=1824]: Waiting for error group managing segments... Jul 26 23:50:46.389429798 (1h59m2.170363167s) [noise=true , address=45.91.116.10:4083 , piece_id=NMWGYAVQVMVXABLZXZ7ENAZNTNWS45HEWBFQRXXSSMQVHNC35JPQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.389975111 (1h59m2.170908472s) [noise=true , address=45.91.116.4:4938 , piece_id=NV3AVDIDY7KAKY73SS4PM6NW2DVDQWJINKKTKAR4FNGBSJDJIXGQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.390112755 (1h59m2.171046131s) [noise=true , address=45.91.116.6:4974 , piece_id=T2HM65HYT7ZMU4U26S3AAEUIZ7W5TCOMQ2I4H4SRAMBSA6SOTBHQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.390469691 (1h59m2.171403055s) [noise=true , address=45.91.116.6:4975 , piece_id=CVZLNWPBWEJYLT4LBFYCBEKYXYH7NOY7GNJPVZW35E7BUPHCBYPQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.390677819 (1h59m2.171611206s) [noise=true , address=45.91.116.8:4047 , piece_id=I6A3I7KVZKDNOFC75R5O24RIH2DQ3XHZPID7QVNKH6ZQHG234DVA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.438128277 (1h59m2.219061658s) [noise=true , address=45.91.116.14:4189 , piece_id=252XE2CKXYGKMMHCACZHVQ4S6MX7YS6IBTFS3BJRJFSI36O2UOTQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:46.503792335 (1h59m2.284725716s) [noise=true , address=45.91.116.9:4262 , piece_id=M5JDD66PKMFZBKFX2JMXASO5FMR6DMXY6WOLDC3MSLIVPUANI77Q, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:47.774058502 (1h59m3.554991865s) [noise=true , address=77.90.31.13:4628 , piece_id=6OIXFOZSSJHKWFHDCIA55FLKIHCC3BQYR7WTDSLX42Q6226FXM7A, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:50:47.793501082 (1h59m3.574434468s) [noise=true , address=77.90.31.15:4697 , piece_id=OG7L57X66JSNP3DO52E6LTCAE4HCBUK3NBPGIOVUYC6QBLEX5OHQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil> Jul 26 23:51:16.235726962 (1h59m32.016660353s) [noise=true , address=104.36.82.16:4294 , piece_id=SGX35GWGTDCM5FIRIGCHEEUOR46DLOU5ZRVMWYN5WVULXDRDGRMQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 159.223.143.39:43906->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.235813169 (1h59m32.016746548s) [noise=true , address=104.36.82.9:4007 , piece_id=XH2DW4QTSPULNSQ2K3DAYU737ZH5VXBSSLCRQBZ7YSWC5ZAMUIMQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:42212->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.235877202 (1h59m32.016810581s) [noise=true , address=104.36.82.2:4732 , piece_id=XMWHE6SDRP7AJZAZVUSXYZQJSR245STM6JPJQL3MSWUNFUF7D7HA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.235943132 (1h59m32.016876512s) [noise=true , address=104.36.82.9:4005 , piece_id=XY4YQ5OI6TOMIZ6ZXDEB4J6VC4P5GRFDLOT7FKIJOYVWJXW6DL6Q, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.236001838 (1h59m32.016935217s) [noise=true , address=104.36.82.13:4187 , piece_id=NCP7XFGLC4HR3JHAURMNVJ3KGOO5ZIC45ZM7447XBQJSG3ETRRNQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:46578->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.236085703 (1h59m32.017019086s) [noise=true , address=104.36.82.5:4839 , piece_id=5RQC5COYAZ5WM3V76UE55QBH3QRCSPNQPX2MT572IOODFHNZ5EYQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.236156367 (1h59m32.017089747s) [noise=true , address=104.36.82.4:4801 , piece_id=A6RPYR5SNYJ3344DBDC4R3RTM3IKQ4DKHJVNFINPI5V6XZD2BVBA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:16.236176291 (1h59m32.017109681s) [seg_pos={0 0}, upload=1824]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:51:16.261018201 (1h59m32.041951593s) [upload=1824]: Done uploading segments... Jul 26 23:51:16.261048963 (1h59m32.041982357s) [upload=1824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.028726733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...X..XX.X...X..X.......X...X....XX..............X.X........] duration: 26.918788473s FAIL (2025-07-26 23:51:08 UTC) context deadline exceeded Duration: 26.918818339s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)