Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XX....X......XX.XXXX...X............................] duration: 642.353636ms FAIL (2025-07-27 01:00:12 UTC) orbiter warning: context deadline exceeded Duration: 642.458681ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX.......] duration: 1.487846433s FAIL (2025-07-27 01:00:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.487883458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................X......X............] duration: 206.186724ms FAIL (2025-07-27 00:59:52 UTC) orbiter warning: uplink: too many requests Duration: 206.22045ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX......] duration: 3.802513404s FAIL (2025-07-27 00:59:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 3.802556338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X.....] duration: 455.520066ms FAIL (2025-07-27 00:59:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.543433ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........XX....X......XX.XXXX...X...........................] duration: 503.034118ms FAIL (2025-07-27 00:59:43 UTC) orbiter warning: context deadline exceeded Duration: 503.061985ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X.....] duration: 486.012107ms FAIL (2025-07-27 00:59:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.04112ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........XX....X......XX.XXXX...X..........................] duration: 465.939122ms FAIL (2025-07-27 00:59:22 UTC) orbiter warning: context deadline exceeded Duration: 465.958873ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...............X...X...............X...........] duration: 1.271478864s FAIL (2025-07-27 00:59:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.272569683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX.....] duration: 902.412952ms FAIL (2025-07-27 00:59:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 902.440784ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...............X...X...............X..........] duration: 4.219407099s FAIL (2025-07-27 00:59:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.219453058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................X......X...........] duration: 399.202585ms FAIL (2025-07-27 00:58:48 UTC) orbiter warning: uplink: too many requests Duration: 399.252082ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...............X...X...............X.........] duration: 962.562713ms FAIL (2025-07-27 00:58:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 962.588527ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X....] duration: 359.595216ms FAIL (2025-07-27 00:58:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 359.625777ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX....] duration: 784.48596ms FAIL (2025-07-27 00:58:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 784.51978ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XX....X......XX.XXXX...X.........................] duration: 545.075349ms FAIL (2025-07-27 00:58:26 UTC) orbiter warning: context deadline exceeded Duration: 545.156021ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X....] duration: 672.947081ms FAIL (2025-07-27 00:58:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 672.983557ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX...] duration: 691.713904ms FAIL (2025-07-27 00:58:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 691.753928ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X...] duration: 379.551465ms FAIL (2025-07-27 00:57:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 379.582575ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................X......X..........] duration: 219.058695ms FAIL (2025-07-27 00:57:37 UTC) orbiter warning: uplink: too many requests Duration: 219.108679ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...............X...X...............X........] duration: 696.533139ms FAIL (2025-07-27 00:57:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 696.61869ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XX....X......XX.XXXX...X........................] duration: 481.929599ms FAIL (2025-07-27 00:57:30 UTC) orbiter warning: context deadline exceeded Duration: 481.96044ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X...] duration: 1.442731957s FAIL (2025-07-27 00:57:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.442780266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X..] duration: 429.24095ms FAIL (2025-07-27 00:57:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.27632ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......................X......X.........] duration: 232.607857ms FAIL (2025-07-27 00:57:00 UTC) orbiter warning: uplink: too many requests Duration: 232.707898ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X..] duration: 361.431085ms FAIL (2025-07-27 00:56:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.481559ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X.] duration: 23.74120378s FAIL (2025-07-27 00:56:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.741229802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX..] duration: 922.269059ms FAIL (2025-07-27 00:56:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 922.297866ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX.] duration: 34.872919475s FAIL (2025-07-27 00:56:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 34.872940852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X.] duration: 21.86165955s FAIL (2025-07-27 00:56:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.861772103s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...............X...X...............X.......] duration: 21.145620355s FAIL (2025-07-27 00:56:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.14563494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......................X......X........] duration: 476.145934ms FAIL (2025-07-27 00:56:34 UTC) orbiter warning: uplink: too many requests Duration: 476.177838ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............XX....X......XX.XXXX...X.......................] duration: 13.639303936s FAIL (2025-07-27 00:56:34 UTC) orbiter warning: context deadline exceeded Duration: 13.639337968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..X] duration: 30.047648712s FAIL (2025-07-27 00:56:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 35.486028007 (3h3m51.266961384s) [noise=true , address=77.90.31.17:4772 , piece_id=CN7XEGZZ5V26NTQEIWXICENWHNAL47EEI6I7SFRKSRQVS5JUXB4Q, seg_pos={0 0}, upload=2841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 159.223.143.39:41830->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:35.486059153 (3h3m51.266992517s) [seg_pos={0 0}, upload=2841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:55:35.486203020 (3h3m51.267136405s) [upload=2841]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:55:35.486210068 (3h3m51.267143452s) [upload=2841]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:55:35.534880589 (3h3m51.315814018s) [upload=2841]: Done uploading segments... Jul 27 00:55:35.534908967 (3h3m51.315842329s) [upload=2841]: Upload finished. err: <nil> Jul 27 00:56:02.927679740 (3h4m18.708613104s) [upload=2853]: Starting upload Jul 27 00:56:02.927758592 (3h4m18.70869196s) [upload=2853]: Done starting upload Jul 27 00:56:02.928146083 (3h4m18.709079452s) [upload=2853]: Uploading segments... Jul 27 00:56:02.928198595 (3h4m18.709131967s) [upload=2853]: Got next segment. Inline: false Jul 27 00:56:02.928237613 (3h4m18.70917098s) [upload=2853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:56:02.928243252 (3h4m18.709176628s) [upload=2853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:56:02.960876085 (3h4m18.741809467s) [seg_pos={0 0}, upload=2853]: Begin upload segment... Jul 27 00:56:02.960963737 (3h4m18.741897105s) [seg_pos={0 0}, upload=2853]: Done begin upload segment. Jul 27 00:56:02.960993910 (3h4m18.741927288s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961027201 (3h4m18.741960591s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961425313 (3h4m18.742358706s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961573671 (3h4m18.742507057s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961829917 (3h4m18.742763302s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.961977104 (3h4m18.742910489s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962202143 (3h4m18.743135523s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962517668 (3h4m18.743451057s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962533722 (3h4m18.743467084s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962592281 (3h4m18.743525667s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.962968364 (3h4m18.743901749s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963091280 (3h4m18.744024667s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963372267 (3h4m18.744305629s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.963472231 (3h4m18.74440561s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964073059 (3h4m18.74500645s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:02.964510517 (3h4m18.745443894s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Uploading piece... Jul 27 00:56:03.065293016 (3h4m18.846226388s) [upload=2853]: Next returned nil segment Jul 27 00:56:03.065318376 (3h4m18.846251763s) [upload=2853]: Waiting for error group managing segments... Jul 27 00:56:03.074437171 (3h4m18.85537054s) [noise=true , address=45.91.116.5:4297 , piece_id=BVMGT5O7MZNIRJUGSREKGATO6DYWAMHL77D5ZBLLOK2JOALGNUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.076435181 (3h4m18.857368549s) [noise=true , address=45.91.116.10:4081 , piece_id=B77E33UN5CM2GPJFY46RSVZI7L3WCCJQ3LQCQ5INSKB4U6K6ZNKQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077056509 (3h4m18.857989875s) [noise=true , address=45.91.116.4:4937 , piece_id=CF7PS35PNLJTAO5J3ZY4XIPOVLEAKO3RJ3IHO6K676XJTROUUJKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077102213 (3h4m18.858035577s) [noise=true , address=45.91.116.7:4009 , piece_id=ZMXNEXZPNKP6WQEHAMQ3R43Z25PUXDK2CEXPBRWURUFNF5CQDEKA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.077805784 (3h4m18.858739167s) [noise=true , address=45.91.116.12:4225 , piece_id=QLNROMCENJHNXKS6X3R6BXQKSK2NXUQVHCOKLP6U2GQGHZQOUUEQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.078723695 (3h4m18.859657073s) [noise=true , address=45.91.116.11:4119 , piece_id=CQ3XQVMD72AVTJIGIHKYCJN3YS7MUCZVUW3FLIBSJRSXHUM73H3A, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.079620481 (3h4m18.860553871s) [noise=true , address=45.91.116.9:4261 , piece_id=4ZTJVOCG4OR2YIZKZBK5YYLB6G5JSWOIL3PUCRZUSP5NW7YG6SLA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.080482721 (3h4m18.861416104s) [noise=true , address=45.91.116.6:4974 , piece_id=P7CFNG75Q56YXXGCRVR7KIXQEM2WCNLR2BUP2DDBAFHCDUFRZBEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:03.105489540 (3h4m18.886422924s) [noise=true , address=45.91.116.14:4189 , piece_id=CUWZATKE5SRIGLNQZWP5LXHIZ3DUQKOMACPOU7K642SKSI3KZOSA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.445384167 (3h4m20.226317551s) [noise=true , address=77.90.31.13:4626 , piece_id=FOGY5WMQQQZLYLJGOYDL6ODHLNXBWEL7MSBPM6UKEKGOXETMWU4Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:04.477516891 (3h4m20.258450256s) [noise=true , address=77.90.31.13:4628 , piece_id=GOH4MJK3IESEJT6IIWZ4NRTZHKFS66SKFF3TYYP33JISQNHAHUTA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:18.884727223 (3h4m34.665660587s) [noise=true , address=77.90.31.15:4697 , piece_id=HHH6FN2ZM2MDQPCU6MY5W5ZF53SQP7QVRYRVUNSEOS2DHW5QG23Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: <nil> Jul 27 00:56:32.896355422 (3h4m48.677288798s) [noise=true , address=104.36.82.5:4838 , piece_id=OWGOF422BDQADI7WHA6RUG3KJW6JP3IIUJRDPLHXDUJ47KTPW2YQ, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896456270 (3h4m48.677389644s) [noise=true , address=104.36.82.7:4911 , piece_id=4YBLBHVNXAWBHOUDYW47RXMWY76KA4Z6E4ZMUHIWVLY7GL52KK5Q, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:43526->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896518005 (3h4m48.677451367s) [noise=true , address=104.36.82.11:4114 , piece_id=YBRSANYH5OAZTBBLTMZKA4FJ6GVENGAM5LUJPG7PVDKL2DNEABEA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896574574 (3h4m48.677507965s) [noise=true , address=104.36.82.14:4222 , piece_id=CO65TR7EBJ2HQ5GLJOLFCFIQWUNVVGFO3QHVY2ENQY3SJUTJ5NDA, seg_pos={0 0}, upload=2853]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:60514->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:32.896637129 (3h4m48.677570496s) [seg_pos={0 0}, upload=2853]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:56:32.941237247 (3h4m48.722170617s) [upload=2853]: Done uploading segments... Jul 27 00:56:32.941261347 (3h4m48.722194724s) [upload=2853]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.047679417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X...............X...X...............X......] duration: 24.184376338s FAIL (2025-07-27 00:56:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.184395406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....X] duration: 30.037796938s FAIL (2025-07-27 00:56:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.037842681s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXXX] duration: 1m0.041598758s FAIL (2025-07-27 00:56:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .880131169 (3h3m32.661064547s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880539501 (3h3m32.661472884s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880662871 (3h3m32.661596256s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.880862469 (3h3m32.66179585s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881161032 (3h3m32.662094415s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881201370 (3h3m32.662134751s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881498014 (3h3m32.662431397s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881598228 (3h3m32.662531614s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.881790195 (3h3m32.662723578s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882039867 (3h3m32.662973257s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882087491 (3h3m32.663020867s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882384654 (3h3m32.663318029s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882436713 (3h3m32.663370097s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.882901415 (3h3m32.66383482s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.887574761 (3h3m32.668508146s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.888195669 (3h3m32.669129062s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.890081919 (3h3m32.671015308s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Uploading piece... Jul 27 00:55:16.965898568 (3h3m32.746831952s) [upload=2835]: Next returned nil segment Jul 27 00:55:16.965919621 (3h3m32.746853005s) [upload=2835]: Waiting for error group managing segments... Jul 27 00:55:17.273720437 (3h3m33.054653806s) [noise=true , address=79.127.201.214:4313 , piece_id=MIJZPWIMFZUR4P47K7MWJW62VKKN2QM47E6X4WI5DDYBXJIFQDBA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.538262514 (3h3m33.319195894s) [noise=true , address=79.127.219.38:4017 , piece_id=6XCZUUXMHSDNIBH2GDZZ56ZKUWCGWGJLO2A5A2JT2532CGAU3ZXQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.544630630 (3h3m33.325564011s) [noise=true , address=79.127.220.101:4041 , piece_id=3PMR7YTCVYVWO5TSS3U4EJIA4W6CJFI55QVO2KJ2S2WLBTDVYR5A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:17.659281529 (3h3m33.440214893s) [noise=true , address=45.91.116.9:4277 , piece_id=4KGEYY2PEOM2RM67XS2AJJPZJDU4MAGU4WSXTUTNNNP2TQ6OXSCQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:20.303749808 (3h3m36.084683187s) [noise=true , address=64.49.13.140:4508 , piece_id=YBGRWUZX3OGA6JTWMKCPFPDXJGJZ465KUCQZ7G2WRHMYSDT52VOA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:22.719100764 (3h3m38.500034151s) [noise=true , address=79.127.201.215:4360 , piece_id=HBOOSFZZSNU5KKUFB3SOM3FFPJBIYDG23MXD7FOWDOCJMNBHC5JA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:55:24.683071569 (3h3m40.464004936s) [noise=true , address=45.91.116.11:4130 , piece_id=GHWOUS6L2NC5AVCG27SX5IL46KZVDGRLDDA54MFOPDA634XKR7GQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:32.741520800 (3h3m48.522454204s) [noise=true , address=64.49.13.137:4502 , piece_id=GFZGEI76DHIGWEVZET7JBEJVZEWGEPLESUSBEZA6OZS2EKVCLW3A, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:03.000543154 (3h4m18.781476538s) [noise=true , address=77.90.31.21:4907 , piece_id=TJHS5HKGMHTROMGTAMG5G7ZZ3LVXL7BZVSKRP2MEDNNAQHWONXOQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:08.860338552 (3h4m24.641271915s) [noise=true , address=77.90.31.12:4616 , piece_id=MXMAZHWFZT6Y5AED2X3H7SZTAR5UP7AL34UPGHFOILQ3QVPBFXLQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: <nil> Jul 27 00:56:16.792492304 (3h4m32.573425686s) [noise=true , address=104.36.82.55:4220 , piece_id=EDRAJQGXAMOWZMG2CNY4UFZ6SS65TZXCM7TEFS2BZBO3KUCH4HEQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:37224->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792568576 (3h4m32.573501953s) [noise=true , address=104.36.82.54:4186 , piece_id=VBRVN4EAHR4PJWYTB6TGEWY56L3M4XNJJ6PMZS4Z6AGZKD7P33EA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 159.223.143.39:33976->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792669830 (3h4m32.573603206s) [noise=true , address=104.36.82.15:4279 , piece_id=4N4A7OHR4BNZRHEWKWG3NQH2CEC5OMGFVFN3EUREMXG32ORXCSKA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792768390 (3h4m32.573701759s) [noise=true , address=104.36.82.53:4132 , piece_id=CSCQCVLQVVHSY5MXR627BWKLIQGIAD7OBNKT7N7W2AUQQPBH5KYA, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792865886 (3h4m32.573799266s) [noise=true , address=104.36.82.47:4538 , piece_id=JIZU65JAZACXQTWRHASJSKPWREQLQCTBVQDCD373KBQTKLRWLCJQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.792969696 (3h4m32.573903057s) [noise=true , address=104.36.82.52:4103 , piece_id=VTEMMSDIS7YMKBEIHN4WGMMLBEF3O7JKOAL34KUPTO7M5ZMWY34Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:56:16.793027791 (3h4m32.57396116s) [noise=true , address=104.36.82.46:4493 , piece_id=RBRLD64I7IRDY44MNOCA7PUAUH3C7USA45DBDEFAYNLBGVL4NU5Q, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793039553 (3h4m32.573972919s) [noise=true , address=104.36.82.14:4256 , piece_id=O77ZG7N56HB5RZUZWS4FUJN3F5XXW3WMLOFTX53XQRFFUESKYXYQ, seg_pos={0 0}, upload=2835]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:56:16.793082172 (3h4m32.574015557s) [seg_pos={0 0}, upload=2835]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:56:16.832048120 (3h4m32.612981485s) [upload=2835]: Done uploading segments... Jul 27 00:56:16.832079329 (3h4m32.613012705s) [upload=2835]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.041614675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X...............X...X...............X.....] duration: 49.402385662s FAIL (2025-07-27 00:56:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.402411368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............XX....X......XX.XXXX...X......................] duration: 25.113119276s FAIL (2025-07-27 00:55:56 UTC) orbiter warning: context deadline exceeded Duration: 25.113175288s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......................X......X.......] duration: 252.805385ms FAIL (2025-07-27 00:55:43 UTC) orbiter warning: uplink: too many requests Duration: 252.846661ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..] duration: 1.858564585s FAIL (2025-07-27 00:55:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 27 00:52:42.873478110 (3h0m58.654411477s) [noise=true , address=104.36.82.9:4005 , piece_id=3FPIL4WOQG3DOFAMQZEP3D5LUQ6NBDWXYDP4JCIPR6S6RMT7NFSQ, seg_pos={0 0}, upload=2798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 159.223.143.39:52664->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:42.873503535 (3h0m58.654436911s) [seg_pos={0 0}, upload=2798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:42.873590035 (3h0m58.6545234s) [upload=2798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:42.873595303 (3h0m58.654528678s) [upload=2798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:42.937011195 (3h0m58.717944581s) [upload=2798]: Done uploading segments... Jul 27 00:52:42.937035418 (3h0m58.717968805s) [upload=2798]: Upload finished. err: <nil> Jul 27 00:53:55.126015126 (3h2m10.90694851s) [upload=2817]: Starting upload Jul 27 00:53:55.126135869 (3h2m10.907069254s) [upload=2817]: Done starting upload Jul 27 00:53:55.126491322 (3h2m10.907424686s) [upload=2817]: Uploading segments... Jul 27 00:53:55.126557628 (3h2m10.907491015s) [upload=2817]: Got next segment. Inline: false Jul 27 00:53:55.126599255 (3h2m10.907532631s) [upload=2817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:55.126605523 (3h2m10.90753891s) [upload=2817]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:55.155376273 (3h2m10.936309657s) [seg_pos={0 0}, upload=2817]: Begin upload segment... Jul 27 00:53:55.155440154 (3h2m10.936373543s) [seg_pos={0 0}, upload=2817]: Done begin upload segment. Jul 27 00:53:55.155470937 (3h2m10.936404323s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.155494226 (3h2m10.936427612s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156059719 (3h2m10.936993104s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156167818 (3h2m10.937101201s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156505619 (3h2m10.937439005s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156562271 (3h2m10.937495653s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156891279 (3h2m10.937824663s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156903943 (3h2m10.937837319s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157282224 (3h2m10.9382156s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157306090 (3h2m10.938239477s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157642347 (3h2m10.938575725s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157707311 (3h2m10.938640697s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158092892 (3h2m10.93902627s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158535071 (3h2m10.939468457s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158987393 (3h2m10.939920779s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.159442456 (3h2m10.940375843s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.253097500 (3h2m11.034030882s) [upload=2817]: Next returned nil segment Jul 27 00:53:55.253118666 (3h2m11.034052055s) [upload=2817]: Waiting for error group managing segments... Jul 27 00:53:55.258249685 (3h2m11.03918309s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258332604 (3h2m11.039265967s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258918124 (3h2m11.0398515s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258975807 (3h2m11.039909184s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259515850 (3h2m11.04044923s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259635850 (3h2m11.040569217s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260090390 (3h2m11.041023766s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260242555 (3h2m11.041175936s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260723294 (3h2m11.041656687s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260867397 (3h2m11.041800775s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.435094721 (3h2m11.216028114s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:56.616631435 (3h2m12.397564818s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:54:25.094965082 (3h2m40.875898466s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:47682->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095080315 (3h2m40.876013703s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:60438->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095113588 (3h2m40.876046958s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095322027 (3h2m40.876255413s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:54:25.095364007 (3h2m40.876297394s) [seg_pos={0 0}, upload=2817]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:25.127534016 (3h2m40.908467396s) [upload=2817]: Done uploading segments... Jul 27 00:54:25.127566856 (3h2m40.908500238s) [upload=2817]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.858602071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXX] duration: 1m0.032139738s FAIL (2025-07-27 00:55:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== :4251 , piece_id=SQW7S5SHTII2OUJU4I3XRVIVOMWSJ5XP4RFWUXO3TVCGYDSWAYPQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.755580071 (3h2m32.536513456s) [noise=true , address=45.91.116.11:4130 , piece_id=IQDMQE7H5YQMIT37UI2EFA5CFHEQNKZV4JS6ABWBLTM6XZNG2RWA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.755739450 (3h2m32.536672827s) [noise=true , address=64.49.13.26:4953 , piece_id=JSJ5BRRJLAIBMZJXYVRASLJKRYMILR7RE63DJGJRPEQFKQIYLWGQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756111287 (3h2m32.537044652s) [noise=true , address=79.127.201.215:4354 , piece_id=PWBILOHFL4LFPEXJRMDD7AI7B5BGRLZDGF2I3GG5JTGHKKVTSPWQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756318393 (3h2m32.53725177s) [noise=true , address=79.127.205.245:4236 , piece_id=KK2NX2OYSBEW2ZFHYNDEAZMDLW3OH3TRZC2QLX65IU5OVGP6LCMQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756429646 (3h2m32.537363082s) [noise=true , address=64.49.13.137:4502 , piece_id=6IJQDAF32P32X4FFBRRACCUGMOZ2L3NQ43QUPBDGHEAR6RWUB3QQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756711422 (3h2m32.537644801s) [noise=true , address=64.49.13.170:4717 , piece_id=VB3GSOCJSB57DZFK33JMJSQVBOQCUIEWBQAL7OXPORQBASZFTHIQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756756090 (3h2m32.537689469s) [noise=true , address=104.36.82.54:4192 , piece_id=7MS5L7BEZDH2QQFAGZNOLKAKFST6JMPRPRD3G65ZDSIEHP7TRKWQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757047069 (3h2m32.537980437s) [noise=true , address=77.90.31.17:4795 , piece_id=T3ZB7QBQCF2FVIFIOCEH2JCXTGGSUZ4JLSRG5GE5QHR724C22X6A, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757218726 (3h2m32.538152103s) [noise=true , address=79.127.205.235:4692 , piece_id=MCGCVQWX5SF2JRLLT5VJCJHIO6ODTO3DWGVGDIOY67ANCU5SWA2Q, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757469923 (3h2m32.538403308s) [noise=true , address=104.36.82.46:4489 , piece_id=SXRMCGBEI6AHONXQNBOZWVIJUTKVHNIMBRYUOQETOZOEQA7FLL5Q, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757864967 (3h2m32.538798346s) [noise=true , address=104.36.82.56:4229 , piece_id=W3KURAVN5NQWYJFEYS2RG7UNDIICXEVSDJAP3QVYAI5WX42Y73HA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.758272858 (3h2m32.539206247s) [noise=true , address=104.36.82.52:4100 , piece_id=A5OHEXUKBOATOOPH6SJN4KLUPOTGHU7YKPKALGRG52XWGG26KUUQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.758656224 (3h2m32.539589606s) [noise=true , address=104.36.82.47:4525 , piece_id=ZGBE436YBZRW2Q6GUCZNMOWKC72Q246R74ZDJXGMNC4TD3HH5OYA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.759053323 (3h2m32.53998672s) [noise=true , address=104.36.82.55:4206 , piece_id=K55J2F4REVE7RUJNSGN5HSJJ2OQSFZCUTNE26DM3XT76OBLUYLZQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.993086636 (3h2m32.774020019s) [upload=2821]: Next returned nil segment Jul 27 00:54:16.993110617 (3h2m32.774044005s) [upload=2821]: Waiting for error group managing segments... Jul 27 00:54:17.374439326 (3h2m33.155372692s) [noise=true , address=79.127.205.235:4692 , piece_id=MCGCVQWX5SF2JRLLT5VJCJHIO6ODTO3DWGVGDIOY67ANCU5SWA2Q, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.434908092 (3h2m33.21584146s) [noise=true , address=64.49.13.170:4717 , piece_id=VB3GSOCJSB57DZFK33JMJSQVBOQCUIEWBQAL7OXPORQBASZFTHIQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.541315587 (3h2m33.322248967s) [noise=true , address=79.127.205.233:4650 , piece_id=BZHEPPYOKFP2IIAACX3BAR6VRU5CEOLHIFFJ5YQN3PAI5CGT3LVQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.787531794 (3h2m33.568465159s) [noise=true , address=64.49.13.137:4502 , piece_id=6IJQDAF32P32X4FFBRRACCUGMOZ2L3NQ43QUPBDGHEAR6RWUB3QQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:18.011681960 (3h2m33.792615325s) [noise=true , address=79.127.205.245:4236 , piece_id=KK2NX2OYSBEW2ZFHYNDEAZMDLW3OH3TRZC2QLX65IU5OVGP6LCMQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:18.306391519 (3h2m34.087324911s) [noise=true , address=64.49.13.26:4953 , piece_id=JSJ5BRRJLAIBMZJXYVRASLJKRYMILR7RE63DJGJRPEQFKQIYLWGQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:20.569130201 (3h2m36.350063586s) [noise=true , address=45.91.116.9:4277 , piece_id=BFHXR3GRRGXFLUDGUHT4SPQDJVYV7BTLTARO3ZI2BD6VB2DLHWWA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:21.251259175 (3h2m37.03219255s) [noise=true , address=45.91.116.11:4130 , piece_id=IQDMQE7H5YQMIT37UI2EFA5CFHEQNKZV4JS6ABWBLTM6XZNG2RWA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:21.924835570 (3h2m37.705768949s) [noise=true , address=79.127.201.215:4354 , piece_id=PWBILOHFL4LFPEXJRMDD7AI7B5BGRLZDGF2I3GG5JTGHKKVTSPWQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:55:16.687076820 (3h3m32.468010212s) [noise=true , address=104.36.82.53:4144 , piece_id=H6GASXUFBNOTUR5GTZ4FC25UNUQP4QHHOTE2ETDTKOFX5HSPCJPQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: rpc: dial tcp 104.36.82.53:4144: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687184443 (3h3m32.468117825s) [noise=true , address=104.36.82.46:4489 , piece_id=SXRMCGBEI6AHONXQNBOZWVIJUTKVHNIMBRYUOQETOZOEQA7FLL5Q, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687248550 (3h3m32.468181926s) [noise=true , address=104.36.82.14:4251 , piece_id=SQW7S5SHTII2OUJU4I3XRVIVOMWSJ5XP4RFWUXO3TVCGYDSWAYPQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: write tcp 159.223.143.39:52606->104.36.82.14:4251: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687508948 (3h3m32.468442315s) [noise=true , address=104.36.82.47:4525 , piece_id=ZGBE436YBZRW2Q6GUCZNMOWKC72Q246R74ZDJXGMNC4TD3HH5OYA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:44592->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687531296 (3h3m32.468464671s) [noise=true , address=104.36.82.56:4229 , piece_id=W3KURAVN5NQWYJFEYS2RG7UNDIICXEVSDJAP3QVYAI5WX42Y73HA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: rpc: dial tcp 104.36.82.56:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687591395 (3h3m32.468524769s) [noise=true , address=104.36.82.52:4100 , piece_id=A5OHEXUKBOATOOPH6SJN4KLUPOTGHU7YKPKALGRG52XWGG26KUUQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687597018 (3h3m32.468530386s) [noise=true , address=104.36.82.55:4206 , piece_id=K55J2F4REVE7RUJNSGN5HSJJ2OQSFZCUTNE26DM3XT76OBLUYLZQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687643383 (3h3m32.468576749s) [noise=true , address=77.90.31.17:4795 , piece_id=T3ZB7QBQCF2FVIFIOCEH2JCXTGGSUZ4JLSRG5GE5QHR724C22X6A, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:49398->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687675572 (3h3m32.468608937s) [noise=true , address=104.36.82.54:4192 , piece_id=7MS5L7BEZDH2QQFAGZNOLKAKFST6JMPRPRD3G65ZDSIEHP7TRKWQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687745470 (3h3m32.46867885s) [seg_pos={0 0}, upload=2821]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 27 00:55:16.716410196 (3h3m32.497343608s) [upload=2821]: Done uploading segments... Jul 27 00:55:16.716442615 (3h3m32.497376s) [upload=2821]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.032161948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....] duration: 505.628511ms FAIL (2025-07-27 00:55:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 505.661512ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......................X......X......] duration: 211.104036ms FAIL (2025-07-27 00:54:45 UTC) orbiter warning: uplink: too many requests Duration: 211.143248ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............XX....X......XX.XXXX...X.....................] duration: 503.302404ms FAIL (2025-07-27 00:54:40 UTC) orbiter warning: context deadline exceeded Duration: 503.337037ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X...............X...X...............X....] duration: 32.430291673s FAIL (2025-07-27 00:54:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.430886695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X.] duration: 424.226169ms FAIL (2025-07-27 00:54:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 27 00:52:42.873478110 (3h0m58.654411477s) [noise=true , address=104.36.82.9:4005 , piece_id=3FPIL4WOQG3DOFAMQZEP3D5LUQ6NBDWXYDP4JCIPR6S6RMT7NFSQ, seg_pos={0 0}, upload=2798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 159.223.143.39:52664->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:42.873503535 (3h0m58.654436911s) [seg_pos={0 0}, upload=2798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:42.873590035 (3h0m58.6545234s) [upload=2798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:42.873595303 (3h0m58.654528678s) [upload=2798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:42.937011195 (3h0m58.717944581s) [upload=2798]: Done uploading segments... Jul 27 00:52:42.937035418 (3h0m58.717968805s) [upload=2798]: Upload finished. err: <nil> Jul 27 00:53:55.126015126 (3h2m10.90694851s) [upload=2817]: Starting upload Jul 27 00:53:55.126135869 (3h2m10.907069254s) [upload=2817]: Done starting upload Jul 27 00:53:55.126491322 (3h2m10.907424686s) [upload=2817]: Uploading segments... Jul 27 00:53:55.126557628 (3h2m10.907491015s) [upload=2817]: Got next segment. Inline: false Jul 27 00:53:55.126599255 (3h2m10.907532631s) [upload=2817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:55.126605523 (3h2m10.90753891s) [upload=2817]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:55.155376273 (3h2m10.936309657s) [seg_pos={0 0}, upload=2817]: Begin upload segment... Jul 27 00:53:55.155440154 (3h2m10.936373543s) [seg_pos={0 0}, upload=2817]: Done begin upload segment. Jul 27 00:53:55.155470937 (3h2m10.936404323s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.155494226 (3h2m10.936427612s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156059719 (3h2m10.936993104s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156167818 (3h2m10.937101201s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156505619 (3h2m10.937439005s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156562271 (3h2m10.937495653s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156891279 (3h2m10.937824663s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156903943 (3h2m10.937837319s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157282224 (3h2m10.9382156s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157306090 (3h2m10.938239477s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157642347 (3h2m10.938575725s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157707311 (3h2m10.938640697s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158092892 (3h2m10.93902627s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158535071 (3h2m10.939468457s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158987393 (3h2m10.939920779s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.159442456 (3h2m10.940375843s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.253097500 (3h2m11.034030882s) [upload=2817]: Next returned nil segment Jul 27 00:53:55.253118666 (3h2m11.034052055s) [upload=2817]: Waiting for error group managing segments... Jul 27 00:53:55.258249685 (3h2m11.03918309s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258332604 (3h2m11.039265967s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258918124 (3h2m11.0398515s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258975807 (3h2m11.039909184s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259515850 (3h2m11.04044923s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259635850 (3h2m11.040569217s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260090390 (3h2m11.041023766s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260242555 (3h2m11.041175936s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260723294 (3h2m11.041656687s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260867397 (3h2m11.041800775s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.435094721 (3h2m11.216028114s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:56.616631435 (3h2m12.397564818s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:54:25.094965082 (3h2m40.875898466s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:47682->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095080315 (3h2m40.876013703s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:60438->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095113588 (3h2m40.876046958s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095322027 (3h2m40.876255413s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:54:25.095364007 (3h2m40.876297394s) [seg_pos={0 0}, upload=2817]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:25.127534016 (3h2m40.908467396s) [upload=2817]: Done uploading segments... Jul 27 00:54:25.127566856 (3h2m40.908500238s) [upload=2817]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.254536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXX] duration: 1m0.028398616s FAIL (2025-07-27 00:54:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3h1m32.393441024s) [upload=2808]: Got next segment. Inline: false Jul 27 00:53:16.612589750 (3h1m32.393523138s) [upload=2808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:16.612597496 (3h1m32.393530889s) [upload=2808]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:16.649453991 (3h1m32.430387387s) [seg_pos={0 0}, upload=2808]: Begin upload segment... Jul 27 00:53:16.649537526 (3h1m32.430470913s) [seg_pos={0 0}, upload=2808]: Done begin upload segment. Jul 27 00:53:16.649568295 (3h1m32.43050168s) [noise=true , address=104.36.82.39:4743 , piece_id=XJ7INQTUC3HXEG7EIXP56SMUA6UBN25K4L27MW6TTEET7A26U2BQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.649584395 (3h1m32.430517781s) [noise=true , address=79.127.205.245:4247 , piece_id=NM42JKIQ5SRNHPULLTBNYZUUWOXGPHNVNUA3FHAYTIYXA5ML7CTQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.650003134 (3h1m32.430936523s) [noise=true , address=79.127.205.235:4685 , piece_id=NARZAGTAYZBJV34FGOXLXO6XWPYYH6LATWBRZDCY22CSK32MKJYQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.650214987 (3h1m32.431148374s) [noise=true , address=79.127.220.101:4041 , piece_id=27NTVODN6LA7B5I2ND3DJEMOBAFNLWMBDNE3KM7SELEPS5RAKQ6Q, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.651266754 (3h1m32.432200135s) [noise=true , address=77.90.31.18:4836 , piece_id=DQ647FBVVXCVQBO4XB2DD4IX7ULLQBPQI3FJ3A72SMT5JNRB37KA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.651653916 (3h1m32.432587297s) [noise=true , address=77.90.31.10:4540 , piece_id=PJBXE3SOHIJCHM2RDBUPEVZKHPOF2ZWUWE5HLBPVBGX4I75PF2CA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652174751 (3h1m32.433108141s) [noise=true , address=104.36.82.47:4545 , piece_id=SPRBQSURNHKI2BD7JLVFUHR27BXFLHL6U6PR4Z7W3VFIVPDELZUQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652578950 (3h1m32.433512362s) [noise=true , address=79.127.201.215:4346 , piece_id=4HJVFIYTRAQE2ZPSBAQCNMWFSRDPDHO6DAXIJQ6VDO7ZDPWVOYLQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652979664 (3h1m32.433913164s) [noise=true , address=79.127.205.234:4620 , piece_id=HZMI5CF7G2LR7FTO2GUK2NIEQFRSG4PYHKM3JEVXTRAHUDT6WNQA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.653885996 (3h1m32.434819382s) [noise=true , address=156.146.43.225:4448 , piece_id=OD5ZLEZV45NXSI5SEY7RXPZCJMZRZM3PGJOQAPPQTALZEM54JRNQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.654332656 (3h1m32.435266041s) [noise=true , address=104.36.82.53:4147 , piece_id=UVQJULEP55JC4J7DZSLTL3BW4EXHWN4UTMDVVFDWLKUXHWCVBJGA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.654731150 (3h1m32.435664537s) [noise=true , address=104.36.82.4:4824 , piece_id=4BKCZFOJ7QRQCHUC7TH6GOSVTS42L7D25W2VDX7QJPZ6SHXLSP2A, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.655250729 (3h1m32.436184114s) [noise=true , address=45.91.116.11:4130 , piece_id=XCGYF3T5O35RESUSB6ANU43GO56BIHT7P7PDUMQVN3MDRF56HG6Q, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.655756052 (3h1m32.436690073s) [noise=true , address=79.127.205.233:4650 , piece_id=SLEMW5GOW3UVFEKJGLFVZLBCX5GY2TAREIMMNFKS5GCQ3ZG4TWBQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.656324680 (3h1m32.437258065s) [noise=true , address=104.36.82.55:4199 , piece_id=7S3U7VJOA7ABYL7I5Y3FPCKLSFJW26U3YF4ZBSZUBXWHLU5IADSA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.656860605 (3h1m32.437794036s) [noise=true , address=104.36.82.46:4489 , piece_id=HMQHDGRC5NJ5FUTFA4BZWWBVFKXSAP3A3CN3XB3HYPUAKOPJAI7A, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.657397946 (3h1m32.438331425s) [noise=true , address=45.91.116.10:4103 , piece_id=ZC7G3BF2HFKXGTMMRZRL2IISDW6HVEEJV7LQ3334VV7ENJB6IJGA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.657883857 (3h1m32.438817241s) [noise=true , address=79.127.205.246:4258 , piece_id=UDV73BKE7HKWHYVDY73PQKYIJU64FVBS4TMECKC7SZARTV4TQUTA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.728714078 (3h1m32.509647453s) [upload=2808]: Next returned nil segment Jul 27 00:53:16.728732317 (3h1m32.509665697s) [upload=2808]: Waiting for error group managing segments... Jul 27 00:53:16.756775898 (3h1m32.537709266s) [noise=true , address=77.90.31.18:4836 , piece_id=DQ647FBVVXCVQBO4XB2DD4IX7ULLQBPQI3FJ3A72SMT5JNRB37KA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:16.860381019 (3h1m32.641314391s) [noise=true , address=77.90.31.10:4540 , piece_id=PJBXE3SOHIJCHM2RDBUPEVZKHPOF2ZWUWE5HLBPVBGX4I75PF2CA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.225416178 (3h1m33.00634962s) [noise=true , address=79.127.205.235:4685 , piece_id=NARZAGTAYZBJV34FGOXLXO6XWPYYH6LATWBRZDCY22CSK32MKJYQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.249550005 (3h1m33.030483383s) [noise=true , address=79.127.205.234:4620 , piece_id=HZMI5CF7G2LR7FTO2GUK2NIEQFRSG4PYHKM3JEVXTRAHUDT6WNQA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.258876279 (3h1m33.039809661s) [noise=true , address=79.127.205.245:4247 , piece_id=NM42JKIQ5SRNHPULLTBNYZUUWOXGPHNVNUA3FHAYTIYXA5ML7CTQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.260264547 (3h1m33.041197925s) [noise=true , address=79.127.205.246:4258 , piece_id=UDV73BKE7HKWHYVDY73PQKYIJU64FVBS4TMECKC7SZARTV4TQUTA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:18.886175063 (3h1m34.667108432s) [noise=true , address=79.127.220.101:4041 , piece_id=27NTVODN6LA7B5I2ND3DJEMOBAFNLWMBDNE3KM7SELEPS5RAKQ6Q, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:19.703702032 (3h1m35.484635417s) [noise=true , address=79.127.205.233:4650 , piece_id=SLEMW5GOW3UVFEKJGLFVZLBCX5GY2TAREIMMNFKS5GCQ3ZG4TWBQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:21.159547004 (3h1m36.94048038s) [noise=true , address=45.91.116.10:4103 , piece_id=ZC7G3BF2HFKXGTMMRZRL2IISDW6HVEEJV7LQ3334VV7ENJB6IJGA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:22.605696066 (3h1m38.386629451s) [noise=true , address=156.146.43.225:4448 , piece_id=OD5ZLEZV45NXSI5SEY7RXPZCJMZRZM3PGJOQAPPQTALZEM54JRNQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:26.226866633 (3h1m42.007800011s) [noise=true , address=45.91.116.11:4130 , piece_id=XCGYF3T5O35RESUSB6ANU43GO56BIHT7P7PDUMQVN3MDRF56HG6Q, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:40.145062262 (3h1m55.925995639s) [noise=true , address=79.127.201.215:4346 , piece_id=4HJVFIYTRAQE2ZPSBAQCNMWFSRDPDHO6DAXIJQ6VDO7ZDPWVOYLQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:54:16.585011327 (3h2m32.365944713s) [noise=true , address=104.36.82.47:4545 , piece_id=SPRBQSURNHKI2BD7JLVFUHR27BXFLHL6U6PR4Z7W3VFIVPDELZUQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585122715 (3h2m32.3660561s) [noise=true , address=104.36.82.55:4199 , piece_id=7S3U7VJOA7ABYL7I5Y3FPCKLSFJW26U3YF4ZBSZUBXWHLU5IADSA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585207431 (3h2m32.366140814s) [noise=true , address=104.36.82.46:4489 , piece_id=HMQHDGRC5NJ5FUTFA4BZWWBVFKXSAP3A3CN3XB3HYPUAKOPJAI7A, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 159.223.143.39:34368->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585457067 (3h2m32.366390447s) [noise=true , address=104.36.82.53:4147 , piece_id=UVQJULEP55JC4J7DZSLTL3BW4EXHWN4UTMDVVFDWLKUXHWCVBJGA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585547307 (3h2m32.366480693s) [noise=true , address=104.36.82.4:4824 , piece_id=4BKCZFOJ7QRQCHUC7TH6GOSVTS42L7D25W2VDX7QJPZ6SHXLSP2A, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585628510 (3h2m32.366561891s) [noise=true , address=104.36.82.39:4743 , piece_id=XJ7INQTUC3HXEG7EIXP56SMUA6UBN25K4L27MW6TTEET7A26U2BQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12XHeYbbHG3TpFDS5ZogLPHZRaYuY7SA5seRU72UoR21JCVcYU9, address:104.36.82.39:4743): protocol: write tcp 159.223.143.39:56318->104.36.82.39:4743: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585703365 (3h2m32.366636753s) [seg_pos={0 0}, upload=2808]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:54:16.607557708 (3h2m32.388491078s) [upload=2808]: Done uploading segments... Jul 27 00:54:16.607582560 (3h2m32.388515927s) [upload=2808]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.028562499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......................X......X.....] duration: 294.986478ms FAIL (2025-07-27 00:54:04 UTC) orbiter warning: uplink: too many requests Duration: 295.018343ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................XX....X......XX.XXXX...X....................] duration: 1.897960778s FAIL (2025-07-27 00:53:59 UTC) orbiter warning: context deadline exceeded Duration: 1.897989451s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)